Recently it was noticed that users on more bleeding-edge Linux distributions that updated saw Easy Anti-Cheat no longer working on Linux, the culprit was glibc and now a Valve developer has spoken out about it.
Writing in a small thread on Twitter, Valve developer Pierre-Loup Griffais said:
Unfortunate that upstream glibc discussion on DT_HASH isn't coming out strongly in favor of prioritizing compatibility with pre-existing applications. Every such instance contributes to damaging the idea of desktop Linux as a viable target for third-party developers.
Our thoughts on the topic from this prior compatibility issue in BlueZ apply more than ever: https://github.com/bluez/bluez/commit/35a2c50437cca4d26ac6537ce3a964bb509c9b62#commitcomment-56028543
It is unfortunately yet another entry in a growing list over the years.We understand that working with a focus on compatibility requires more resources and more engineering trade-offs, but strongly believe it is nonetheless the way to go. We are very interested in helping with any underlying resource constraints.
This prompted CodeWeavers (who work on Wine and with Valve on Proton) developer Arek Hiler to write a blog post titled "Win32 Is The Only Stable ABI on Linux" and their ending statement is something people should think on:
I think this whole situation shows why creating native games for Linux is challenging. It’s hard to blame developers for targeting Windows and relying on Wine + friends. It’s just much more stable and much less likely to break and stay broken.
Hiler certainly isn't the only one to think like that, with another CodeWeavers developer Andrew Eikum mentioning on Hacker News some time ago:
As a long-time Linux dev (see my profile), I have also found this to be true. Linux userland APIs are unstable and change all the time. Some transitions that come to mind that have affected me personally: ALSA->pulse; libudev->libudev2->systemd; gstreamer 0.10->1.0. All of those changes required modifications to my software, and the backwards-compat tools that are provided are buggy and insufficient. Meanwhile, you can still write and run winmm[1] applications on Windows 10, and they will work in almost all cases. It's simply the case that the win32 API is more stable than Linux userland APIs, so it's entirely plausible that games will run better in Wine, which shares that stable ABI, than they will on Linux, especially as time goes on and Linux userland shifts yet again.
[1] winmm dates to the Windows 3.x days!
Situations like this can be pretty messy and this is not a case of open source versus secret closed source anti-cheat stuff either, since the glibc issue affected a Native Linux game (Shovel Knight) and Linux software libstrangle. No doubt there are other things yet to be discovered that were broken by the change.
It is of course also a case that Linux distributions need to ensure they do quality assurance testing, especially for gaming which can end up showing up issues quite easily and that bleeding-edge distributions can and clearly do end up breaking things by pulling new software in so quickly.
Article taken from GamingOnLinux.com.
Some you may have missed, popular articles from the last month:
The comments on this article are closed.
Quoting: EagleDeltaQuoting: BlackBloodRumIt's all on usage case. For a server, stable is better.
What is the definition of "stable" here? For things like RHEL, CentOS, Debian Stable, SLES, sometimes even Ubuntu Server "stable" generally means very, very old versions of software that many times is no longer maintained by the upstream developers. In many cases when I was a SysAdmin, that meant we had to install external YUM or APT repos to install the software that the project or company "supported" because what was included in the distribution was no longer supported by the community or the company behind the FOSS project and as such, support was entirely on us to manage.
As for enterprise, the advent of patterns like containerization (Docker, containerd, Flatpak, etc) has enabled the ability to have a stable OS with newer (or older) libraries running inside the application "container". However, in the case of glibc, I'm not sure that's an option. Especially on Desktops. glibc is so core to the OS that the container would have to ship GFX drivers as well that are built on the shipped glibc version separate from the host OS and anything else that entails.
When it comes to core system functionality, compatibility will always win out from the non-technical customers/users. There's a reason Microsoft has not successfully moved away from win32 yet, despite the amount of times they've wanted to and tried.
Stable means that the API/ABI are unchanged for the lifecycle of the distribution so that enterprises can build software for say RHEL7 and their customers can then be sure that this software will work exactly identical on RHEL7.0 as it will on RHEL7.9 even though there are 5 years between the two releases.
While the included software might be versions that are no longer being maintained upstream they are being maintained by the distribution developers (this is why a subscription to RHEL does not come cheap) and security features are backported, which is why even if it might look like you run PHP v3.4 you do infact run a highly patched version of PHP that is just as secure as PHP 8.x but walks and talks like 3.4
That you had to import external repositories with newer software was a decision made by your company developers to use that newer software instead of the one provided by the distro of choice. Either the devs have to be kept in check (they will always want the new shiny) or the choice of distribution have to match what they need/want, the kind of combination that your workplace tried is usually a road to unnecessary complexity and problems.
0 Likes
Quoting: JordanPlayz158Quoting: F.UltraYeah, I tried that then I thought `man elf` and there is a page, is this from glibc's documentation or something else as if they have control over that man page then they should definitely change this portionQuoting: JordanPlayz158Quoting: F.UltraWell I just had one example experience (with xdg-desktop-portal, apologizes if they are an exception and not the norm for the standards of C libraries), I try to stay away from languages like C just as it doesn't really seem to click with me like a lot of other languages have (not to mention there are languages like Rust that some say will replace C and while I don't like rust either, it is leagues better for clicking with me). Also yeah, that makes a lot more sense as I was very confused by DT_HASH being in a build config file and not a function in the code when I saw the commit so again, thank you for clearing that up.Quoting: JordanPlayz158Quoting: ShabbyXIt's made exactly so that libraries *can* break ABI if they have to, without the world imploding. glibc made a breaking change, and no matter how small, they should have made an incompatible version change. Yes that would still be inconvenient, but at least it's detectable and fixable. Imagine if python3 did all its backward incompatible things but still called itself python2.Yes, that's exactly the point we were making, they didn't treat it as a breaking change as they just incremented the patch number by 1 rather than the major. Not to mention a lot of people have said it wasn't very clear and I could buy that as documentation doesn't seem to be a strong point in either linux or c in general (or I could be wrong (or xdg-desktop-portal could be a bad example of a c library) but when I attempted to make an application using xdg-desktop-portal they had the equivalent of a javadoc but that was about it, there are no examples provided in the repo for how to use any of the objects (or it wasn't clearly labeled or findable), there were no tutorials online and I looked into other repos (that are flatpaks as they must use xdg-desktop-portal to my knowledge) and it was a mess because different applications use different languages or sometimes used a middle man library or some combination of the two which made it difficult to know how to use the interface)
glibc is very well documented, as is most C libraries on Linux (don't know where this misconception comes from), they didn't see the ELF sections as part of the glibc ABI/API though which is why this went under the radar for developers that isn't on the glibc mailing lists.
This was not a code change, it was a change in the automake configure script to no longer pass in the argument to ld to include the old hash section when building glibc.
Normally no library/application dev ever sees a change like this as a ABI/API change, but it looks like the glibc devs will do from now on.
no probs, xdg-desktop-portal looks to be a 3d party tool for flatpak and unfortunately many such well intended extra tools are very rarely well documented.
To experience the glibc documentation just open a terminal and type in say "man dlopen" and see just how much information they have on that single function.
d_tag This member may have any of the following values:
DT_NULL Marks end of dynamic section
DT_NEEDED String table offset to name of a needed library
DT_PLTRELSZ Size in bytes of PLT relocation entries
DT_PLTGOT Address of PLT and/or GOT
DT_HASH Address of symbol hash table
to
d_tag This member may have any of the following values:
...
DT_HASH [s]Address of symbol hash table[/s] PLEASE FOR THE LOVE OF GOD DO NOT USE THIS AND INSTEAD USE DT_GNU_HASH IN THE FUTURE, PLEASE USE ANYTHING OTHER THAN THIS
and then problem solved but it is possible this man page is not in their control but if it is, I propose to add my addition to it lol
Yes the elf man page looks to come from not glibc since it does not cover DT_GNU_HASH, it looks to be from the kernel and have a git page here: https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git and I can see that DT_GNU_HASH is still not in the elf(5) page in MAIN/HEAD.
However I guess that this could be problematic on another level as well since not only is glibc not the only libc out there for Linux but it is also not the only shared object and this man page covers the ELF format itself and not just the one supported by the glibc ld binary.
0 Likes
Quoting: F.UltraQuoting: JordanPlayz158Quoting: minidouI fail to see how glibc devs could be blamed for removing a function deprecated for 17 years (though I can agree a major version bump should have been called for). Do videogame devs need that much nursing ? EAC linux lib is a recent piece of software, how many deprecated dependencies do they use ?Quoting: TermyI tend to agree with the criticism about unstable APIs for many cases - but in this case, i really blame Epic for using a function deprecated for almost two decades in a recent piece of software.From everything I've read from other people, it wasn't obvious that the function was depreciated. You shouldn't need to search up, "is x function depreciated glibc" for every function you use, it should be immediately obvious (like most languages I know have comments or a depreciation tag), some say that all distros somehow knew about this but you shouldn't need to be familiar with linux development or need to be in the right communities to know about a depreciated function.
Small nitpick here, but DT_HASH is not a libc function, it's a linker section in the ELF header of a shared object. The function that utilizes this section is dlopen() and that function have not changed and it works, it's only applications that tries to read the ELF data manually that have to handle DT_HASH and DT_GNU_HASH, and by all means it looks like the glibc devs "incorrectly" thought that dlopen was their ABI/API and not the actual linker section in the ELF headers.
I find this neither nitpcking nor small! So it isn't an actual API change, but considered an internal data invisible to almost every user program.
0 Likes
Quoting: shorbergI wasn't going to comment on this because it is way off-topic, but having thought about it for a while I decided to do so anyway, in the interest of public education on neural diversity. Liam or moderators may remove this if they think it inappropriate.
Quoting: dibzYeah, one of the best and worst things about Linux is the vocal community. And let's be honest, the people that flock tend to have very strong personalities -- borderline autistic at times.Autistic is not the word you are looking for here. It is a common and widespread misconception, but it really isn't what you are looking for to describe the individuals you are hinting at. Autistics are often asocial, generally not antisocial. Typically, if an autistic person learns they have said or done something that have hurt someone, they will feel intense remorse.
A "borderline autistic" person is a perfectly average person, you likely won't even notice anything different except perhaps slightly increased anxiety compared to baseline human their age. At most you might describe them as someone who likes to stay home and read a book on a Saturday every now and then.
They are not people who will try to rally the masses, they will probably just stand at the sideline wishing they could contribute more. There are exceptions though, a certain fellow countrywoman is a strong personality who does quite a bit to hold people accountable for the climate.
Quoting wikipedia,
QuoteThe autism spectrum is a range of neurodevelopmental conditions generally characterized by difficulties in social interactions and communication, repetitive behaviours, intense interests, and unusual responses to sensory stimuli.The difficulty in social interaction and communication refers to discomfort and avoidance and reduced body language. But many autistics won't put themselves in most social situations to begin with, either because of a lack of interest in social activities or because it confuses them. Even mailing-lists, or forum posting is something that scares many.
Source: I've done a lot of reading on autism, I have talked to a lot of professionals, I have friends who are autistic and I am autistic myself. I am however far from an expert, and open to learn more.
PS. What I have described as autistics above is not a set of all people with autism spectrum disorder, it is a generalised set of the ones who are high-functioning enough that you are likely to have been in contact with them in something as complex as technical online discussions.
I appreciate the explanations. Though, my use of "Borderline Autistic" is actually close to home for me. Without airing my laundry on the internet, I will say that I used the term in the context that I learned it in the first place -- in a healthcare setting when diagnosing one of my children who I could absolutely see being a "strongly opinionated linux user" like many of us, heh.
1 Likes, Who?
Quoting: EikeQuoting: F.UltraQuoting: JordanPlayz158Quoting: minidouI fail to see how glibc devs could be blamed for removing a function deprecated for 17 years (though I can agree a major version bump should have been called for). Do videogame devs need that much nursing ? EAC linux lib is a recent piece of software, how many deprecated dependencies do they use ?Quoting: TermyI tend to agree with the criticism about unstable APIs for many cases - but in this case, i really blame Epic for using a function deprecated for almost two decades in a recent piece of software.From everything I've read from other people, it wasn't obvious that the function was depreciated. You shouldn't need to search up, "is x function depreciated glibc" for every function you use, it should be immediately obvious (like most languages I know have comments or a depreciation tag), some say that all distros somehow knew about this but you shouldn't need to be familiar with linux development or need to be in the right communities to know about a depreciated function.
Small nitpick here, but DT_HASH is not a libc function, it's a linker section in the ELF header of a shared object. The function that utilizes this section is dlopen() and that function have not changed and it works, it's only applications that tries to read the ELF data manually that have to handle DT_HASH and DT_GNU_HASH, and by all means it looks like the glibc devs "incorrectly" thought that dlopen was their ABI/API and not the actual linker section in the ELF headers.
I find this neither nitpcking nor small! So it isn't an actual API change, but considered an internal data invisible to almost every user program.
Well it's complicated :). The glibc devs obviously didn't consider this a part of their API/ABI (they might do so going forward however) of their library. On the other hand ELF itself can from a point of view be considered to be the ABI of a shared object.
99.99999% of all applications does not care one cent about any of the ELF headers, all the work of parsing out the data to be used is handled by the libc functions such as dlopen() but some special applications, and EAC does match this criteria, wants to read/write and parse the ELF data directly and not use any of the provided libc functions. For EAC they do this in case dlopen() have been patched under their nose to hide the fact that this .so loads a cheating mod.
The problem here then is that the official ELF spec only contains DH_HASH and then have some mumble mumble about possible extensions while glibc back in 2005/2006 introduced DH_GNU_HASH as the better alternative. The official ELF specs does not carry "vendor extensions" AFAIK, of which this is one, and the glibc devs didn't really care either since they made sure that gcc and ld could handle both (and later clang, musl etc implemented the very same since they saw that gcc and ld handled this extension).
So we can't really blame the EAC devs either more to the point that they should perhaps have not just read the specs but also taken a look at how the actual implementation (gcc+ld) works since ELF can have extensions and as I wrote earlier I wonder if this could open a potential hole in the current implementation of EAC where a maleficent .so creates a nice looking DH_HASH while hiding their real stuff in DH_GNU_HASH since EAC only looks at the first while ld loads the second (but I'm not intimate enough with the exact details here to know for sure, perhaps ld loads only used DH_GNU_HASH if DH_HASH isn't there or EAC loads the .so for the game so that ld isn't involved at all).
What I do think that we can blame is all the people that scream bloody murder, I mean yes it was a problem, but it was fixed and it was fixed quickly. That last part is IMHO the important part and what makes Linux really shine, so to me this should be seen more as a positive thing and not as the horrible negative thing that it have became.
3 Likes, Who?
CyborgZeta Aug 18, 2022
Thank for the responses to my earlier question.
Can I just give credit to the Arch Linux developers/maintainers for quickly spotting this issue and patching it? They sure handled this promptly.
Can I just give credit to the Arch Linux developers/maintainers for quickly spotting this issue and patching it? They sure handled this promptly.
2 Likes, Who?
PublicNuisance Aug 19, 2022
So an employee from a company who makes a closed source game store client that hosts a closed source game that uses closed source anticheat software had an issue with an open source library and i'm supposed to side with the employee ? Do I about have that right ? Would it not be an easier issue to fix if more of that equation I listed were open source ?
0 Likes
Cloversheen Aug 19, 2022
Quoting: dibzI appreciate the explanations. Though, my use of "Borderline Autistic" is actually close to home for me. Without airing my laundry on the internet, I will say that I used the term in the context that I learned it in the first place -- in a healthcare setting when diagnosing one of my children who I could absolutely see being a "strongly opinionated linux user" like many of us, heh.
Fair!
If you feel it appropriate and you think your child would like it, say "Hi!" from me. You do not have to state your decision on this on the public forum.
1 Likes, Who?
Quoting: PublicNuisanceSo an employee from a company who makes a closed source game store client that hosts a closed source game that uses closed source anticheat software had an issue with an open source library and i'm supposed to side with the employee ? Do I about have that right ? Would it not be an easier issue to fix if more of that equation I listed were open source ?It wasn't just EAC that broke but a few other applications like Shovel Knight, a game with a native linux port and libstrangle, an fps limiting library, this isn't about open vs closed source software, this is about maintaining backwards compatibility to not only make linux more enticing to port to but also to make it so things like games that do not get maintained forever won't get lost to time.
0 Likes
slaapliedje Aug 19, 2022
Quoting: JordanPlayz158Well, it 'sort of' is about open vs closed. If it was open (Shovel Knight) someone could have given them a pull request to change to the newer method. That's pretty straight forward.Quoting: PublicNuisanceSo an employee from a company who makes a closed source game store client that hosts a closed source game that uses closed source anticheat software had an issue with an open source library and i'm supposed to side with the employee ? Do I about have that right ? Would it not be an easier issue to fix if more of that equation I listed were open source ?It wasn't just EAC that broke but a few other applications like Shovel Knight, a game with a native linux port and libstrangle, an fps limiting library, this isn't about open vs closed source software, this is about maintaining backwards compatibility to not only make linux more enticing to port to but also to make it so things like games that do not get maintained forever won't get lost to time.
This wasn't really about backward compatibility, and more about deprecated methods still being used in software. This should be a wake up call for those projects that use old crappy code, and hopefully they'll get updated to be faster.
0 Likes
Quoting: slaapliedjeWe'll have to agree to disagree on that front, most of the outrage I've seen is about glibc developers not making it clear they were removing DT_HASH (and that you needed to go to mailing lists and such to know it was depreciated, incremented the patch rather than the major for a breaking change) and I'd say the bigger deal of hurting the prospect of the Linux desktop being ported to or getting more native applications (especially for game developers when games inevitably stop getting maintained and typically do not get their source released once the game is EOL) due to these sort of changesQuoting: JordanPlayz158Well, it 'sort of' is about open vs closed. If it was open (Shovel Knight) someone could have given them a pull request to change to the newer method. That's pretty straight forward.Quoting: PublicNuisanceSo an employee from a company who makes a closed source game store client that hosts a closed source game that uses closed source anticheat software had an issue with an open source library and i'm supposed to side with the employee ? Do I about have that right ? Would it not be an easier issue to fix if more of that equation I listed were open source ?It wasn't just EAC that broke but a few other applications like Shovel Knight, a game with a native linux port and libstrangle, an fps limiting library, this isn't about open vs closed source software, this is about maintaining backwards compatibility to not only make linux more enticing to port to but also to make it so things like games that do not get maintained forever won't get lost to time.
This wasn't really about backward compatibility, and more about deprecated methods still being used in software. This should be a wake up call for those projects that use old crappy code, and hopefully they'll get updated to be faster.
0 Likes
Quoting: JordanPlayz158Quoting: slaapliedjeWe'll have to agree to disagree on that front, most of the outrage I've seen is about glibc developers not making it clear they were removing DT_HASH (and that you needed to go to mailing lists and such to know it was depreciated, incremented the patch rather than the major for a breaking change) and I'd say the bigger deal of hurting the prospect of the Linux desktop being ported to or getting more native applications (especially for game developers when games inevitably stop getting maintained and typically do not get their source released once the game is EOL) due to these sort of changesQuoting: JordanPlayz158Well, it 'sort of' is about open vs closed. If it was open (Shovel Knight) someone could have given them a pull request to change to the newer method. That's pretty straight forward.Quoting: PublicNuisanceSo an employee from a company who makes a closed source game store client that hosts a closed source game that uses closed source anticheat software had an issue with an open source library and i'm supposed to side with the employee ? Do I about have that right ? Would it not be an easier issue to fix if more of that equation I listed were open source ?It wasn't just EAC that broke but a few other applications like Shovel Knight, a game with a native linux port and libstrangle, an fps limiting library, this isn't about open vs closed source software, this is about maintaining backwards compatibility to not only make linux more enticing to port to but also to make it so things like games that do not get maintained forever won't get lost to time.
This wasn't really about backward compatibility, and more about deprecated methods still being used in software. This should be a wake up call for those projects that use old crappy code, and hopefully they'll get updated to be faster.
This was certainly a documentation mistake, but as well misunderstanding what they (glibc) consider as their api/abi, and what some applications did consider by it.
That said, it should have been a wakeup call to those using it, and we will probably see ports to DT_GNU_HASH, and DT_HASH will disappear with a major release at some point.
That will give them thoughts about what they want to deprecate of the ELF sections too for a next major release. And it's good this process started.
It's a real nieche usage, I personally never did require to use DT_HASH or DT_GNU_HASH since dlopen pretty much did work for my cases. But I undetstand wanting to directly read library information in some use cases (security scanners being one, they certainly don't want something messing with dlopen, and I know some commercial tools messing with the dlopen and generally library loading process, dynatrace comes to mind as an example).
Last edited by STiAT on 19 August 2022 at 7:00 pm UTC
1 Likes, Who?
Kuduzkehpan Aug 19, 2022
Print("""
no native computing?
still no?
why should we expect rather than this?
""")
native = input("Native Computing Or not Choose wisely and case sensitive")
if native != "Native Computing":
print("you deserved that")
else
print("Finally May the open source be with you")
no native computing?
still no?
why should we expect rather than this?
""")
native = input("Native Computing Or not Choose wisely and case sensitive")
if native != "Native Computing":
print("you deserved that")
else
print("Finally May the open source be with you")
0 Likes
Quoting: KlaasQuoting: KuduzkehpanelseSyntaxError: expected ':'
and what's that? Python? lol... the langauge best known to break API in minor versions and even bugfix releases. By honest mistakes which got fixed with fast follow up releases, but still. Got better in the past few years though, was a lot worse in 2.x, but happened in 3.x too (3.6 series in example).
Last edited by STiAT on 20 August 2022 at 1:35 am UTC
0 Likes
slaapliedje Aug 20, 2022
Quoting: STiATBash is notorious for this as well.Quoting: KlaasQuoting: KuduzkehpanelseSyntaxError: expected ':'
and what's that? Python? lol... the langauge best known to break API in minor versions and even bugfix releases. By honest mistakes which got fixed with fast follow up releases, but still. Got better in the past few years though, was a lot worse in 2.x, but happened in 3.x too (3.6 series in example).
So in a case like this with glibc, did they need to yell from the rooftops that they were now going to remove the 'both' from the automake file? Maybe post it on Twitter, Facebook, CNN? I'm honestly curious how something like this gets stated so people know that something that was added has deprecated older bits, and you should no longer use them. This is longer ago than a huge majority of people even know what Linux was...
0 Likes
Quoting: JordanPlayz158Quoting: PublicNuisanceSo an employee from a company who makes a closed source game store client that hosts a closed source game that uses closed source anticheat software had an issue with an open source library and i'm supposed to side with the employee ? Do I about have that right ? Would it not be an easier issue to fix if more of that equation I listed were open source ?It wasn't just EAC that broke but a few other applications like Shovel Knight, a game with a native linux port and libstrangle, an fps limiting library, this isn't about open vs closed source software, this is about maintaining backwards compatibility to not only make linux more enticing to port to but also to make it so things like games that do not get maintained forever won't get lost to time.
More to the point, I understand why EAC cares about the internal representation, but I don't understand the usecase of shovel knight and libstrangle. What they do with the hash table that can't be accomplished with the standard API?
1 Likes, Who?
Patreon
PayPal
See more from me