Recently it was noticed that users on more bleeding-edge Linux distributions that updated saw Easy Anti-Cheat no longer working on Linux, the culprit was glibc and now a Valve developer has spoken out about it.
Writing in a small thread on Twitter, Valve developer Pierre-Loup Griffais said:
Unfortunate that upstream glibc discussion on DT_HASH isn't coming out strongly in favor of prioritizing compatibility with pre-existing applications. Every such instance contributes to damaging the idea of desktop Linux as a viable target for third-party developers.
Our thoughts on the topic from this prior compatibility issue in BlueZ apply more than ever: https://github.com/bluez/bluez/commit/35a2c50437cca4d26ac6537ce3a964bb509c9b62#commitcomment-56028543
It is unfortunately yet another entry in a growing list over the years.We understand that working with a focus on compatibility requires more resources and more engineering trade-offs, but strongly believe it is nonetheless the way to go. We are very interested in helping with any underlying resource constraints.
This prompted CodeWeavers (who work on Wine and with Valve on Proton) developer Arek Hiler to write a blog post titled "Win32 Is The Only Stable ABI on Linux" and their ending statement is something people should think on:
I think this whole situation shows why creating native games for Linux is challenging. It’s hard to blame developers for targeting Windows and relying on Wine + friends. It’s just much more stable and much less likely to break and stay broken.
Hiler certainly isn't the only one to think like that, with another CodeWeavers developer Andrew Eikum mentioning on Hacker News some time ago:
As a long-time Linux dev (see my profile), I have also found this to be true. Linux userland APIs are unstable and change all the time. Some transitions that come to mind that have affected me personally: ALSA->pulse; libudev->libudev2->systemd; gstreamer 0.10->1.0. All of those changes required modifications to my software, and the backwards-compat tools that are provided are buggy and insufficient. Meanwhile, you can still write and run winmm[1] applications on Windows 10, and they will work in almost all cases. It's simply the case that the win32 API is more stable than Linux userland APIs, so it's entirely plausible that games will run better in Wine, which shares that stable ABI, than they will on Linux, especially as time goes on and Linux userland shifts yet again.
[1] winmm dates to the Windows 3.x days!
Situations like this can be pretty messy and this is not a case of open source versus secret closed source anti-cheat stuff either, since the glibc issue affected a Native Linux game (Shovel Knight) and Linux software libstrangle. No doubt there are other things yet to be discovered that were broken by the change.
It is of course also a case that Linux distributions need to ensure they do quality assurance testing, especially for gaming which can end up showing up issues quite easily and that bleeding-edge distributions can and clearly do end up breaking things by pulling new software in so quickly.
Quoting: BlackBloodRumMy point was that people (such as yourself) often believed that enterprise level distros were pointless even for servers (…)That's really insulting and bordering on defamation. Please don't put words in my mouth that I'd never say.
Quoting: CyborgZetaBut wouldn't running games, and programs/applications in general, inside containers fix this kind of issue? Isn't that one of the goals of Flatpak?
You wouldn't be able to use things like the nvidia graphics driver anymore unless you'd include a matching version as well. And then you'd have to fix the kernel.
Last edited by Klaas on 18 August 2022 at 12:03 am UTC
Quoting: GuestThere's an easy solution, ban the use of EAC in GNU+Linux native games.
Oh, wait...
Except that Shovel Knight is a native linux game that doesn't use EAC and it's still broken by the glibc update.
I think the glibc devs need to be more careful about the kind of updates they push and the impact they can have.
Quoting: Zagorimhttps://www.phoronix.com/forums/forum/software/linux-gaming/1340528-glibc-2-36-dropping-dt_hash-has-been-breaking-easy-anti-cheat-games-with-steam-playQuoting: GuestThere's an easy solution, ban the use of EAC in GNU+Linux native games.
Oh, wait...
Except that Shovel Knight is a native linux game that doesn't use EAC and it's still broken by the glibc update.
I think the glibc devs need to be more careful about the kind of updates they push and the impact they can have.
So according to this article and comments...
The superior method for for which broke (DT_HASH) is to use DT_GNU_HASH. Which has been around for 16 years...
Just to be fair to Linux and all the 'no stable ABI!' This is the GNU library, and isn't controlled at all by the Linux kernel, though clearly just as important.
So this isn't a case of the glibc devs needing to bw careful. They literally just changed the default compile flags to no longer include both methods, because they figured (clearly incorrectly) that people had all shifted to the much more optimized method. Which they have had 16 years to do.
Also sounds like Arch Linux at least has already changed their compile flags and people are testing 2.36-2 which seems to fix the EAC issues (and presumably games like Shovel Knight).
Edit: more technical details here.
https://flapenguin.me/elf-dt-gnu-hash
Sounds to me like some things could end up with massive performance boosts, if it was using the better prelinking!
Last edited by slaapliedje on 18 August 2022 at 9:11 am UTC
Quoting: BlackBloodRumIt's all on usage case. For a server, stable is better.
What is the definition of "stable" here? For things like RHEL, CentOS, Debian Stable, SLES, sometimes even Ubuntu Server "stable" generally means very, very old versions of software that many times is no longer maintained by the upstream developers. In many cases when I was a SysAdmin, that meant we had to install external YUM or APT repos to install the software that the project or company "supported" because what was included in the distribution was no longer supported by the community or the company behind the FOSS project and as such, support was entirely on us to manage.
As for enterprise, the advent of patterns like containerization (Docker, containerd, Flatpak, etc) has enabled the ability to have a stable OS with newer (or older) libraries running inside the application "container". However, in the case of glibc, I'm not sure that's an option. Especially on Desktops. glibc is so core to the OS that the container would have to ship GFX drivers as well that are built on the shipped glibc version separate from the host OS and anything else that entails.
When it comes to core system functionality, compatibility will always win out from the non-technical customers/users. There's a reason Microsoft has not successfully moved away from win32 yet, despite the amount of times they've wanted to and tried.
QuoteMeanwhile, you can still write and run winmm[1] applications on Windows 10, and they will work in almost all cases
For some reason it's impossible to reply to the hackernews thread where this originally came from. Anyway, to that I would reply: Then please, write software with WINMM.
In an open-source setting (project), when a system library changes behavior (such as in the bluez case), downstream code can be adapted. Conversely, because adaptation is possible, such library changes are not seen as big a blocker as otherwise. It's a "good vicious circle", and it means that such systems need little, if any, layers of compatibility/emulation, which should benefit performance. Closed-source binaries just don't fit into that ethos, and (for lim time->\infinity), all closed-source binaries will need some emulation *anyway* for one reason or another, so there's little incentive for open-source projects to bend around for a closed executable just so it can run without compat layers for a few months.
So by all means, do write all closed software for WINMM.
Quoting: BeamboomQuoting: F.UltraNew releases of WINE/Proton breaks games that worked with the older version from time to time so yes they are wrong in that WIN32 is the only stable ABI on Linux since it clearly isn't stable.
It's not a either/or, binary reality discussed here. It's not a question if this or that ever breaks at all, ever.
So how am I then to interpret CodeWeavers "WIN32 is the only stable ABI on Linux" when it have broken more times for me on my Linux system than glibc have ever done, yet this change by glibc triggers CodeWeavers to make that tweet?!
Quoting: minidouI fail to see how glibc devs could be blamed for removing a function deprecated for 17 years (though I can agree a major version bump should have been called for). Do videogame devs need that much nursing ? EAC linux lib is a recent piece of software, how many deprecated dependencies do they use ?
Quoting: TermyI tend to agree with the criticism about unstable APIs for many cases - but in this case, i really blame Epic for using a function deprecated for almost two decades in a recent piece of software.From everything I've read from other people, it wasn't obvious that the function was depreciated. You shouldn't need to search up, "is x function depreciated glibc" for every function you use, it should be immediately obvious (like most languages I know have comments or a depreciation tag), some say that all distros somehow knew about this but you shouldn't need to be familiar with linux development or need to be in the right communities to know about a depreciated function.
Quoting: KlaasBackporting security fixes can lead to a lot of trouble if something is missed. Remember the Debian OpenSSL bug a few years ago?
That had nothing to do with backporting, it was the Debian maintainer thinking that he could improve the entropy of the OpenSSL PRNG with a quick hack of his own.
Quoting: ZagorimQuoting: GuestThere's an easy solution, ban the use of EAC in GNU+Linux native games.
Oh, wait...
Except that Shovel Knight is a native linux game that doesn't use EAC and it's still broken by the glibc update.
I think the glibc devs need to be more careful about the kind of updates they push and the impact they can have.
The question is why ShovelKnight have the same problem, very very few applications (and none of them games) should ever have the reason the read the ELF data manually. Everyone else should use the libc provided dlopen() function to open shared dynamic libraries.
Does ShovelKnight use some EAC like middleware?
Quoting: ShabbyXIt's made exactly so that libraries *can* break ABI if they have to, without the world imploding. glibc made a breaking change, and no matter how small, they should have made an incompatible version change. Yes that would still be inconvenient, but at least it's detectable and fixable. Imagine if python3 did all its backward incompatible things but still called itself python2.Yes, that's exactly the point we were making, they didn't treat it as a breaking change as they just incremented the patch number by 1 rather than the major. Not to mention a lot of people have said it wasn't very clear and I could buy that as documentation doesn't seem to be a strong point in either linux or c in general (or I could be wrong (or xdg-desktop-portal could be a bad example of a c library) but when I attempted to make an application using xdg-desktop-portal they had the equivalent of a javadoc but that was about it, there are no examples provided in the repo for how to use any of the objects (or it wasn't clearly labeled or findable), there were no tutorials online and I looked into other repos (that are flatpaks as they must use xdg-desktop-portal to my knowledge) and it was a mess because different applications use different languages or sometimes used a middle man library or some combination of the two which made it difficult to know how to use the interface)
Quoting: JordanPlayz158Quoting: minidouI fail to see how glibc devs could be blamed for removing a function deprecated for 17 years (though I can agree a major version bump should have been called for). Do videogame devs need that much nursing ? EAC linux lib is a recent piece of software, how many deprecated dependencies do they use ?Quoting: TermyI tend to agree with the criticism about unstable APIs for many cases - but in this case, i really blame Epic for using a function deprecated for almost two decades in a recent piece of software.From everything I've read from other people, it wasn't obvious that the function was depreciated. You shouldn't need to search up, "is x function depreciated glibc" for every function you use, it should be immediately obvious (like most languages I know have comments or a depreciation tag), some say that all distros somehow knew about this but you shouldn't need to be familiar with linux development or need to be in the right communities to know about a depreciated function.
Small nitpick here, but DT_HASH is not a libc function, it's a linker section in the ELF header of a shared object. The function that utilizes this section is dlopen() and that function have not changed and it works, it's only applications that tries to read the ELF data manually that have to handle DT_HASH and DT_GNU_HASH, and by all means it looks like the glibc devs "incorrectly" thought that dlopen was their ABI/API and not the actual linker section in the ELF headers.
Quoting: F.UltraOf course the Codeweavers dev would say that. It's kind of their business to sell Win32 compatibility to Linux users. I thought I'd be nice and throw some money their way for the mac version of Crossover, but for the most part I've found newer wine is just more capable.Quoting: BeamboomQuoting: F.UltraNew releases of WINE/Proton breaks games that worked with the older version from time to time so yes they are wrong in that WIN32 is the only stable ABI on Linux since it clearly isn't stable.
It's not a either/or, binary reality discussed here. It's not a question if this or that ever breaks at all, ever.
So how am I then to interpret CodeWeavers "WIN32 is the only stable ABI on Linux" when it have broken more times for me on my Linux system than glibc have ever done, yet this change by glibc triggers CodeWeavers to make that tweet?!
Quoting: JordanPlayz158Quoting: ShabbyXIt's made exactly so that libraries *can* break ABI if they have to, without the world imploding. glibc made a breaking change, and no matter how small, they should have made an incompatible version change. Yes that would still be inconvenient, but at least it's detectable and fixable. Imagine if python3 did all its backward incompatible things but still called itself python2.Yes, that's exactly the point we were making, they didn't treat it as a breaking change as they just incremented the patch number by 1 rather than the major. Not to mention a lot of people have said it wasn't very clear and I could buy that as documentation doesn't seem to be a strong point in either linux or c in general (or I could be wrong (or xdg-desktop-portal could be a bad example of a c library) but when I attempted to make an application using xdg-desktop-portal they had the equivalent of a javadoc but that was about it, there are no examples provided in the repo for how to use any of the objects (or it wasn't clearly labeled or findable), there were no tutorials online and I looked into other repos (that are flatpaks as they must use xdg-desktop-portal to my knowledge) and it was a mess because different applications use different languages or sometimes used a middle man library or some combination of the two which made it difficult to know how to use the interface)
glibc is very well documented, as is most C libraries on Linux (don't know where this misconception comes from), they didn't see the ELF sections as part of the glibc ABI/API though which is why this went under the radar for developers that isn't on the glibc mailing lists.
This was not a code change, it was a change in the automake configure script to no longer pass in the argument to ld to include the old hash section when building glibc.
Normally no library/application dev ever sees a change like this as a ABI/API change, but it looks like the glibc devs will do from now on.
Quoting: F.UltraThank you for correcting the information, I am not a C developer nor have made linux applications "natively" so thank you for clearing that upQuoting: JordanPlayz158Quoting: minidouI fail to see how glibc devs could be blamed for removing a function deprecated for 17 years (though I can agree a major version bump should have been called for). Do videogame devs need that much nursing ? EAC linux lib is a recent piece of software, how many deprecated dependencies do they use ?Quoting: TermyI tend to agree with the criticism about unstable APIs for many cases - but in this case, i really blame Epic for using a function deprecated for almost two decades in a recent piece of software.From everything I've read from other people, it wasn't obvious that the function was depreciated. You shouldn't need to search up, "is x function depreciated glibc" for every function you use, it should be immediately obvious (like most languages I know have comments or a depreciation tag), some say that all distros somehow knew about this but you shouldn't need to be familiar with linux development or need to be in the right communities to know about a depreciated function.
Small nitpick here, but DT_HASH is not a libc function, it's a linker section in the ELF header of a shared object. The function that utilizes this section is dlopen() and that function have not changed and it works, it's only applications that tries to read the ELF data manually that have to handle DT_HASH and DT_GNU_HASH, and by all means it looks like the glibc devs "incorrectly" thought that dlopen was their ABI/API and not the actual linker section in the ELF headers.
Quoting: F.UltraWell I just had one example experience (with xdg-desktop-portal, apologizes if they are an exception and not the norm for the standards of C libraries), I try to stay away from languages like C just as it doesn't really seem to click with me like a lot of other languages have (not to mention there are languages like Rust that some say will replace C and while I don't like rust either, it is leagues better for clicking with me). Also yeah, that makes a lot more sense as I was very confused by DT_HASH being in a build config file and not a function in the code when I saw the commit so again, thank you for clearing that up.Quoting: JordanPlayz158Quoting: ShabbyXIt's made exactly so that libraries *can* break ABI if they have to, without the world imploding. glibc made a breaking change, and no matter how small, they should have made an incompatible version change. Yes that would still be inconvenient, but at least it's detectable and fixable. Imagine if python3 did all its backward incompatible things but still called itself python2.Yes, that's exactly the point we were making, they didn't treat it as a breaking change as they just incremented the patch number by 1 rather than the major. Not to mention a lot of people have said it wasn't very clear and I could buy that as documentation doesn't seem to be a strong point in either linux or c in general (or I could be wrong (or xdg-desktop-portal could be a bad example of a c library) but when I attempted to make an application using xdg-desktop-portal they had the equivalent of a javadoc but that was about it, there are no examples provided in the repo for how to use any of the objects (or it wasn't clearly labeled or findable), there were no tutorials online and I looked into other repos (that are flatpaks as they must use xdg-desktop-portal to my knowledge) and it was a mess because different applications use different languages or sometimes used a middle man library or some combination of the two which made it difficult to know how to use the interface)
glibc is very well documented, as is most C libraries on Linux (don't know where this misconception comes from), they didn't see the ELF sections as part of the glibc ABI/API though which is why this went under the radar for developers that isn't on the glibc mailing lists.
This was not a code change, it was a change in the automake configure script to no longer pass in the argument to ld to include the old hash section when building glibc.
Normally no library/application dev ever sees a change like this as a ABI/API change, but it looks like the glibc devs will do from now on.
Quoting: slaapliedjeQuoting: F.UltraOf course the Codeweavers dev would say that. It's kind of their business to sell Win32 compatibility to Linux users. I thought I'd be nice and throw some money their way for the mac version of Crossover, but for the most part I've found newer wine is just more capable.Quoting: BeamboomQuoting: F.UltraNew releases of WINE/Proton breaks games that worked with the older version from time to time so yes they are wrong in that WIN32 is the only stable ABI on Linux since it clearly isn't stable.
It's not a either/or, binary reality discussed here. It's not a question if this or that ever breaks at all, ever.
So how am I then to interpret CodeWeavers "WIN32 is the only stable ABI on Linux" when it have broken more times for me on my Linux system than glibc have ever done, yet this change by glibc triggers CodeWeavers to make that tweet?!
I know, it just bothers me that there are whole reddit threads now filled with (very outspoken) people that believe them. Basically this whole thing is nothing with a storm in a glass, for a short period there existed a breakage for people using bleeding edge rolling distros, but thanks to this all being FOSS those distros could quickly patch the DT_HASH section back in and later glibc themselves fixed it upstream (yes they where initially a bit grumpy about it, which is understandably considering that they did it not for laughs but for actual benefits and have been waiting 16 years to do it. I also think that people put way to much emphasis on devs being grumpy, part of a devs job is to reject patches and being grumpy, otherwise things like glibc would be several GB by now).
Quoting: JordanPlayz158Quoting: F.UltraWell I just had one example experience (with xdg-desktop-portal, apologizes if they are an exception and not the norm for the standards of C libraries), I try to stay away from languages like C just as it doesn't really seem to click with me like a lot of other languages have (not to mention there are languages like Rust that some say will replace C and while I don't like rust either, it is leagues better for clicking with me). Also yeah, that makes a lot more sense as I was very confused by DT_HASH being in a build config file and not a function in the code when I saw the commit so again, thank you for clearing that up.Quoting: JordanPlayz158Quoting: ShabbyXIt's made exactly so that libraries *can* break ABI if they have to, without the world imploding. glibc made a breaking change, and no matter how small, they should have made an incompatible version change. Yes that would still be inconvenient, but at least it's detectable and fixable. Imagine if python3 did all its backward incompatible things but still called itself python2.Yes, that's exactly the point we were making, they didn't treat it as a breaking change as they just incremented the patch number by 1 rather than the major. Not to mention a lot of people have said it wasn't very clear and I could buy that as documentation doesn't seem to be a strong point in either linux or c in general (or I could be wrong (or xdg-desktop-portal could be a bad example of a c library) but when I attempted to make an application using xdg-desktop-portal they had the equivalent of a javadoc but that was about it, there are no examples provided in the repo for how to use any of the objects (or it wasn't clearly labeled or findable), there were no tutorials online and I looked into other repos (that are flatpaks as they must use xdg-desktop-portal to my knowledge) and it was a mess because different applications use different languages or sometimes used a middle man library or some combination of the two which made it difficult to know how to use the interface)
glibc is very well documented, as is most C libraries on Linux (don't know where this misconception comes from), they didn't see the ELF sections as part of the glibc ABI/API though which is why this went under the radar for developers that isn't on the glibc mailing lists.
This was not a code change, it was a change in the automake configure script to no longer pass in the argument to ld to include the old hash section when building glibc.
Normally no library/application dev ever sees a change like this as a ABI/API change, but it looks like the glibc devs will do from now on.
no probs, xdg-desktop-portal looks to be a 3d party tool for flatpak and unfortunately many such well intended extra tools are very rarely well documented.
To experience the glibc documentation just open a terminal and type in say "man dlopen" and see just how much information they have on that single function.
Quoting: F.UltraExactly this! It's a compile flag. Nothing more.Quoting: JordanPlayz158Quoting: ShabbyXIt's made exactly so that libraries *can* break ABI if they have to, without the world imploding. glibc made a breaking change, and no matter how small, they should have made an incompatible version change. Yes that would still be inconvenient, but at least it's detectable and fixable. Imagine if python3 did all its backward incompatible things but still called itself python2.Yes, that's exactly the point we were making, they didn't treat it as a breaking change as they just incremented the patch number by 1 rather than the major. Not to mention a lot of people have said it wasn't very clear and I could buy that as documentation doesn't seem to be a strong point in either linux or c in general (or I could be wrong (or xdg-desktop-portal could be a bad example of a c library) but when I attempted to make an application using xdg-desktop-portal they had the equivalent of a javadoc but that was about it, there are no examples provided in the repo for how to use any of the objects (or it wasn't clearly labeled or findable), there were no tutorials online and I looked into other repos (that are flatpaks as they must use xdg-desktop-portal to my knowledge) and it was a mess because different applications use different languages or sometimes used a middle man library or some combination of the two which made it difficult to know how to use the interface)
glibc is very well documented, as is most C libraries on Linux (don't know where this misconception comes from), they didn't see the ELF sections as part of the glibc ABI/API though which is why this went under the radar for developers that isn't on the glibc mailing lists.
This was not a code change, it was a change in the automake configure script to no longer pass in the argument to ld to include the old hash section when building glibc.
Normally no library/application dev ever sees a change like this as a ABI/API change, but it looks like the glibc devs will do from now on.
Quoting: F.UltraYeah, I tried that then I thought `man elf` and there is a page, is this from glibc's documentation or something else as if they have control over that man page then they should definitely change this portionQuoting: JordanPlayz158Quoting: F.UltraWell I just had one example experience (with xdg-desktop-portal, apologizes if they are an exception and not the norm for the standards of C libraries), I try to stay away from languages like C just as it doesn't really seem to click with me like a lot of other languages have (not to mention there are languages like Rust that some say will replace C and while I don't like rust either, it is leagues better for clicking with me). Also yeah, that makes a lot more sense as I was very confused by DT_HASH being in a build config file and not a function in the code when I saw the commit so again, thank you for clearing that up.Quoting: JordanPlayz158Quoting: ShabbyXIt's made exactly so that libraries *can* break ABI if they have to, without the world imploding. glibc made a breaking change, and no matter how small, they should have made an incompatible version change. Yes that would still be inconvenient, but at least it's detectable and fixable. Imagine if python3 did all its backward incompatible things but still called itself python2.Yes, that's exactly the point we were making, they didn't treat it as a breaking change as they just incremented the patch number by 1 rather than the major. Not to mention a lot of people have said it wasn't very clear and I could buy that as documentation doesn't seem to be a strong point in either linux or c in general (or I could be wrong (or xdg-desktop-portal could be a bad example of a c library) but when I attempted to make an application using xdg-desktop-portal they had the equivalent of a javadoc but that was about it, there are no examples provided in the repo for how to use any of the objects (or it wasn't clearly labeled or findable), there were no tutorials online and I looked into other repos (that are flatpaks as they must use xdg-desktop-portal to my knowledge) and it was a mess because different applications use different languages or sometimes used a middle man library or some combination of the two which made it difficult to know how to use the interface)
glibc is very well documented, as is most C libraries on Linux (don't know where this misconception comes from), they didn't see the ELF sections as part of the glibc ABI/API though which is why this went under the radar for developers that isn't on the glibc mailing lists.
This was not a code change, it was a change in the automake configure script to no longer pass in the argument to ld to include the old hash section when building glibc.
Normally no library/application dev ever sees a change like this as a ABI/API change, but it looks like the glibc devs will do from now on.
no probs, xdg-desktop-portal looks to be a 3d party tool for flatpak and unfortunately many such well intended extra tools are very rarely well documented.
To experience the glibc documentation just open a terminal and type in say "man dlopen" and see just how much information they have on that single function.
d_tag This member may have any of the following values:
DT_NULL Marks end of dynamic section
DT_NEEDED String table offset to name of a needed library
DT_PLTRELSZ Size in bytes of PLT relocation entries
DT_PLTGOT Address of PLT and/or GOT
DT_HASH Address of symbol hash table
to
d_tag This member may have any of the following values:
...
DT_HASH [s]Address of symbol hash table[/s] PLEASE FOR THE LOVE OF GOD DO NOT USE THIS AND INSTEAD USE DT_GNU_HASH IN THE FUTURE, PLEASE USE ANYTHING OTHER THAN THIS
and then problem solved but it is possible this man page is not in their control but if it is, I propose to add my addition to it lol
See more from me