Recently it was noticed that users on more bleeding-edge Linux distributions that updated saw Easy Anti-Cheat no longer working on Linux, the culprit was glibc and now a Valve developer has spoken out about it.
Writing in a small thread on Twitter, Valve developer Pierre-Loup Griffais said:
Unfortunate that upstream glibc discussion on DT_HASH isn't coming out strongly in favor of prioritizing compatibility with pre-existing applications. Every such instance contributes to damaging the idea of desktop Linux as a viable target for third-party developers.
Our thoughts on the topic from this prior compatibility issue in BlueZ apply more than ever: https://github.com/bluez/bluez/commit/35a2c50437cca4d26ac6537ce3a964bb509c9b62#commitcomment-56028543
It is unfortunately yet another entry in a growing list over the years.We understand that working with a focus on compatibility requires more resources and more engineering trade-offs, but strongly believe it is nonetheless the way to go. We are very interested in helping with any underlying resource constraints.
This prompted CodeWeavers (who work on Wine and with Valve on Proton) developer Arek Hiler to write a blog post titled "Win32 Is The Only Stable ABI on Linux" and their ending statement is something people should think on:
I think this whole situation shows why creating native games for Linux is challenging. It’s hard to blame developers for targeting Windows and relying on Wine + friends. It’s just much more stable and much less likely to break and stay broken.
Hiler certainly isn't the only one to think like that, with another CodeWeavers developer Andrew Eikum mentioning on Hacker News some time ago:
As a long-time Linux dev (see my profile), I have also found this to be true. Linux userland APIs are unstable and change all the time. Some transitions that come to mind that have affected me personally: ALSA->pulse; libudev->libudev2->systemd; gstreamer 0.10->1.0. All of those changes required modifications to my software, and the backwards-compat tools that are provided are buggy and insufficient. Meanwhile, you can still write and run winmm[1] applications on Windows 10, and they will work in almost all cases. It's simply the case that the win32 API is more stable than Linux userland APIs, so it's entirely plausible that games will run better in Wine, which shares that stable ABI, than they will on Linux, especially as time goes on and Linux userland shifts yet again.
[1] winmm dates to the Windows 3.x days!
Situations like this can be pretty messy and this is not a case of open source versus secret closed source anti-cheat stuff either, since the glibc issue affected a Native Linux game (Shovel Knight) and Linux software libstrangle. No doubt there are other things yet to be discovered that were broken by the change.
It is of course also a case that Linux distributions need to ensure they do quality assurance testing, especially for gaming which can end up showing up issues quite easily and that bleeding-edge distributions can and clearly do end up breaking things by pulling new software in so quickly.
Article taken from GamingOnLinux.com.
Some you may have missed, popular articles from the last month:
The comments on this article are closed.
Quoting: GuestThe blame is not really about removing the deprecated feature. Blame is for the failure to restore it after discovering that users for the feature still exist and the removal broke stuff.Yes, a lot of developers (especially newer, younger ones that lack the experience) argue in favor of always using the very latest everything and always keeping everything updated.
I've seen that time and time again in several teams.
But that's not a position that can be maintained in reality.
You'd require armies just to maintain old stuff.
It's fine to remove a function if you can't see that anyone is using it. An understandable mistake.
But the moment you realize that there were indeed many still using it, it should clearly be restored (with a deprecation marker, but still).
The only exception here would be security vulnerabilities. Those breaking software is even desirable.
10 Likes, Who?
Quoting: BeamboomQuoting: 1xokThis whole criticism from CodeWeavers developers should be taken with a pinch of salt...... Why?
Because WINE is where their money comes from?
It's not like they're a neutral party in this question...
2 Likes, Who?
Quoting: EikeQuoting: BeamboomQuoting: 1xokThis whole criticism from CodeWeavers developers should be taken with a pinch of salt...... Why?
Because WINE is where their money comes from?
It's not like they're a neutral party in this question...
I see nothing wrong or biased in what they said.
5 Likes, Who?
Quoting: GuestQuoting: minidouI fail to see how glibc devs could be blamed for removing a function deprecated for 17 years (though I can agree a major version bump should have been called for). Do videogame devs need that much nursing ? EAC linux lib is a recent piece of software, how many deprecated dependencies do they use ?The blame is not really about removing the deprecated feature. Blame is for the failure to restore it after discovering that users for the feature still exist and the removal broke stuff.
They did restore it because of the backlash, but they really shouldn't have, there's no reason for it. This isn't some obscure lib we're talking about but the GNU C library. If a handful of program gets broken it's on them, the function was marked deprecated almost two decades ago. An outstanding majority of program have no issue at all with the removal of this function, why should a niche but maybe high visibility user dictate the direction of glibc ?
Something not discussed here is that the removal of DT_HASH allows a save of about 1% or 16kB of space per Glibc shared object. This is an improvement. There clearly is a good reason for finally removing this deprecated function. But we shouldn't profit from this improvement because of some devs bad practice ?
Quoting: TheSHEEEPQuoting: GuestThe blame is not really about removing the deprecated feature. Blame is for the failure to restore it after discovering that users for the feature still exist and the removal broke stuff.Yes, a lot of developers (especially newer, younger ones that lack the experience) argue in favor of always using the very latest everything and always keeping everything updated.
I've seen that time and time again in several teams.
But that's not a position that can be maintained in reality.
You'd require armies just to maintain old stuff.
It's fine to remove a function if you can't see that anyone is using it. An understandable mistake.
This is off topic. The impacted softwares were initialised years after DT_HASH was marked deprecated. In the case of EAC lib, probably a good 15 years later. In every sector of the software industry this would be considered bad practice, but VG softwares get a pass for some reason.
Quoting: TheSHEEEPBut the moment you realize that there were indeed many still using it, it should clearly be restored (with a deprecation marker, but still).
They did that, almost two decades ago.
And it seems there isn't that many users impacted. I can count the EAC lib and some lib used in shovel night. Compared to the millions of programs using glibc.
3 Likes, Who?
Quoting: BeamboomQuoting: EikeQuoting: BeamboomQuoting: 1xokThis whole criticism from CodeWeavers developers should be taken with a pinch of salt...... Why?
Because WINE is where their money comes from?
It's not like they're a neutral party in this question...
I see nothing wrong or biased in what they said.
We didn't say it's wrong. But they are biased. It's hard to take an absolute neutral standpoint about something that feeds your family. This doesn't automatically mean there's something wrong with what they said. Just have your salt ready.
But, as you asked:
QuoteIt’s hard to blame developers for targeting Windows and relying on Wine + friends. It’s just much more stable and much less likely to break and stay broken.
I have heard it's not that easy and automatically working to run old Windows games on Windows, so I am not sure about this. I would even think that Code Weavers have to work around such changes. What I positively know (as they are mentioning changes in Linux sound interfaces) is that my Sound Blaster card stopped working on Windows with some update and continued to work on Linux.
1 Likes, Who?
dziadulewicz Aug 17, 2022
Quoting: JpxeIsn't Flatpak the solution to this? Then you can just target the runtime environment and it will always stay the same
Or Snap rather. Snaps are much more versatile than flatpaks. They are not each others drop in replacements against a surprisingly common misconception.
Packagers in general very often choose Snap because Flatpak is made for desktop applications only, while Snap can ship libraries, apps, server/CLI software, and much more.
The longer initial startup time of Firefox is most definitely not a reasonable excuse to suggest ditching the whole thing which is a work in progress anyways.
1 Likes, Who?
Quoting: minidouSomething not discussed here is that the removal of DT_HASH allows a save of about 1% or 16kB of space per Glibc shared object. This is an improvement. There clearly is a good reason for finally removing this deprecated function. But we shouldn't profit from this improvement because of some devs bad practice ?That's neat.
But irrelevant when compared to the importance of not breaking user space.
You could add a megabyte to each object and it would still pale in comparison to how important not breaking existing software is.
Memory is not an issue anymore nowadays outside of very specific environments. We've left the 90s a few decades ago.
And those specific environments can compile their own glibc with the flag enabling the optimization if they need it. In fact, optimizations like that are standard for low-spec environments.
Quoting: minidouThey did that, almost two decades ago.You are not thinking things through.
And it seems there isn't that many users impacted. I can count the EAC lib and some lib used in shovel night. Compared to the millions of programs using glibc.
This is just one library.
Imagine if all libraries took this approach. A few pieces of software here using something deprecated from lib X, a few pieces of software there using something deprecated from lib Y...
And very, very soon you will have tons of stuff not working anymore (especially older software that isn't maintained, but still used by some or even many).
If MS followed that approach, Windows wouldn't have anywhere near the desktop share it has now.
They are clearly doing something very, very right with maintaining legacy code.
Also, so what if they did that 20 years ago?
Somehow, EAC and a bunch of others still ended up using that function. Probably looked something up online, copied the function, it worked and that's it. Or had auto-complete suggest something, or whatever.
You can't expect programmers to double-check every single function they use for possible deprecation, that's absurd.
Deprecation warnings in compilations get overlooked, too.
That's not good, you can call it bad practice or whatever makes you feel good about yourself, but in practice, it happens. A lot. And that's the reality that has to be dealt with, not some kind of utopia in which programmers have eradicated bad practice...
Last edited by TheSHEEEP on 17 August 2022 at 11:42 am UTC
9 Likes, Who?
Quoting: EikeWe didn't say it's wrong. But they are biased. It's hard to take an absolute neutral standpoint about something that feeds your family. This doesn't automatically mean there's something wrong with what they said. Just have your salt ready.
In real life absolute objectivity/neutrality doesn't exist. If you have insight into something, you'll always have some sort of alignment. You come from somewhere who has some sort of connection to the topic.
But what's said here really folds neatly into what's been said by multiple sources basically since the beginning of Linux. There's simply no reason to doubt (or take with a pinch of salt) what's here said. Even if you work as a developer for a given firm, doesn't mean you *always* will talk like a marketeer.
Some even use it as an argument *for* Linux, as the backward compatibility of Windows really is a nightmare for them - it adds overhead and/or complexity to the platform. Getting rid of obsolete calls is also a luxury.
A *lot* of the security breaches over the years has been rooted in the chaos of backward compatibility.
That's another fun factor here: Everything has their pros and cons. :)
Last edited by Beamboom on 18 August 2022 at 10:29 am UTC
2 Likes, Who?
QuoteBut irrelevant when compared to the importance of not breaking user space.
Nothing got broken. A two decade of depcrecation function got removed, but nothing broken. Or do we just expect everything to be forever maintained ?
Quoting: TheSHEEEPAlso, so what if they did that 20 years ago?
Somehow, EAC and a bunch of others still ended up using that function. Probably looked something up online, copied the function, it worked and that's it. Or had auto-complete suggest something, or whatever.
You can't expect programmers to double-check every single function they use for possible deprecation, that's absurd.
I don't expect anyone to check, I expect a CI or a quality gate to stop them from shipping.
Quoting: TheSHEEEPDeprecation warnings in compilations get overlooked.
That's not good, you can call it bad practice or whatever makes you feel good about yourself, but in practice, it happens. A lot. And that's the reality that has to be dealt with, not some kind utopia in which programmers have eradicated bad practice...
I'll call it bad practice, or just not being up to 2022 standards.
0 Likes
glibc compatibility has been a pain for a while.
I have released 2 games on Steam with native Linux ports.
I'm always afraid of doing a dist-upgrade, because it's very likely the compiled binaries won't run on the previous distro version.
It's absurd, but targeting win32 is more stable.
There should be a way to have multiple working glibc versions.
I have released 2 games on Steam with native Linux ports.
I'm always afraid of doing a dist-upgrade, because it's very likely the compiled binaries won't run on the previous distro version.
It's absurd, but targeting win32 is more stable.
There should be a way to have multiple working glibc versions.
11 Likes, Who?
Quoting: minidouNothing got broken. A two decade of depcrecation function got removed, but nothing broken.Really?
Well, then I guess this is all just a bunch of fake news.
Quoting: minidouOr do we just expect everything to be forever maintained ?As long as people use it? Optimally, yes.
Ever heard of COBOL?
Quoting: minidouI don't expect anyone to check, I expect a CI or a quality gate to stop them from shipping.Oh, you sweet little summer child...
Quoting: minidouI'll call it bad practice, or just not being up to 2022 standards.Welcome to the real world.
Where libraries have to deal with not being used correctly without breaking the software that is using them incorrectly...
Where ages-old code has to be kept around or otherwise "bad things happen(tm)".
And where most people don't have the luxury of time to make everything work perfectly, and so mistakes happen.
13 Likes, Who?
Quoting: TheSHEEEPMemory is not an issue anymore nowadays outside of very specific environments.
This is absolutely not true. 16KB is 4 pages of memory, saving that on every .so is huge! It's not just that you have the memory laying around, there are other costs too. There's the cost of loading the objects from disk, maintaining the struct page entries in the kernel etc.
There is a reason Linux is _fast_. With your approach, Linux would have been bloatware like the rest of them.
Quoting: TheSHEEEPThis is just one library.
Imagine if all libraries took this approach.
There is something called semantic versioning: https://en.m.wikipedia.org/wiki/Software_versioning#Semantic_versioning
It's made exactly so that libraries *can* break ABI if they have to, without the world imploding. glibc made a breaking change, and no matter how small, they should have made an incompatible version change. Yes that would still be inconvenient, but at least it's detectable and fixable. Imagine if python3 did all its backward incompatible things but still called itself python2.
Also, I don't understand at all how anyone could be defending win32 here. win32 is a piece of hot garbage. No one likes is, no one wants it, not even Microsoft. Is anyone suggesting we should stick with some shitty API for another 1000 years because ABIs should never change?
No, software changes, it's the nature of it (soft is in its name!). And ABIs break when they need to. But you have versions just so you can deal with this.
Last edited by ShabbyX on 18 August 2022 at 3:40 am UTC
5 Likes, Who?
People seem to forget what Arch, Tumbleweed and other bleeding edge rolling distros are. With bleeding edge you have to expect that API and ABIs are removed after a certain time of deprecation. If you want stability for many years use a regular release LTS distribution like Leap or RockyLinux/CentOS. It's not like Microsoft didn't break anything between Windows 7/10/11.
3 Likes, Who?
Quoting: ShabbyXThere is a reason Linux is _fast_.Linux was fast before DT_HASH got removed.
I very seriously doubt the improvement gained by that is even noticeable for most people.
I'm not saying you can be generally messy with memory, unaware of alignment, etc.
But this one change was not some incredible performance boost that would've made a breaking change "worthwhile".
Quoting: ShabbyXWith your approach, Linux would have been bloatware like the rest of them.With my approach, Linux would be as fast as it is, while also much less prone to breaking and much more attractive to people not liking their stuff break every now and again.
I don't mind it so much myself, it's generally easy to fix, too.
But that's not the mindset of most people.
Quoting: ShabbyXIt's made exactly so that libraries *can* break ABI if they have to, without the world imploding. glibc made a breaking change, and no matter how small, they should have made an incompatible version change. Yes that would still be inconvenient, but at least it's detectable and fixable. Imagine if python3 did all it's backward incompatible things but still called itself python2.You are right, but we talking about what glibc DID do, not what they SHOULD have done.
And this isn't the first time something like that happened, either, outside of major version changes.
Of course I am assuming that major versions break things, that's just normal.
3 Likes, Who?
Quoting: GuestQuoting: ShabbyXThis is absolutely not true. 16KB is 4 pages of memory, saving that on every .so is huge! It's not just that you have the memroy laying around, there are other costs too. There's the cost of loading the objects from disk, maintaing the struct page entries in the kernel etc.Everything has worked fine so far even with this extra cost, so I doubt the (real world) effect is huge.
Many small changes can sum up to a huge effect, though.
Quoting: GuestWhat kind of improvement this change makes for desktop use case?
Glibc is not only targeted at desktop use case.
1 Likes, Who?
CyborgZeta Aug 17, 2022
Quoting: orzoI've seen this video before. It's certainly aged well.Quoting: ZlopezEven Linus Torvalds said: "Never break user space!"Here is his full speech (+/- 6 minutes), including rant on glibc. It's eight years old now.
https://youtu.be/5PmHRSeA2c8?t=333
1 Likes, Who?
Quoting: TheSHEEEPAlso, so what if they did that 20 years ago?
Somehow, EAC and a bunch of others still ended up using that function. Probably looked something up online, copied the function, it worked and that's it. Or had auto-complete suggest something, or whatever.
You can't expect programmers to double-check every single function they use for possible deprecation, that's absurd.
Oh that's easy to explain the DT_GNU_HASH function isn't as well documented according to what several devs have said. To the point where I've seen several point out some aspects of it are basically only documented via the mailing list and that's just bad look no matter how you slice it when talking bout something as core as a C library.
9 Likes, Who?
I don't understand why people think that it is a good idea to break existing binary only applications and at the same time expecting that games get native releases. Those two thing are completely incompatible. If you want the Linux desktop to be able to gain a larger market share and get any official releases from commercial developers, such breakages are not an option. It's the same discussion as dropping multilib support since it “only takes space”.
Note: I do not care about EAC breakage (except on principle) since I refuse to play games that use it, but I'm sure that Shovel Knight is not the only native release that is affected by this change.
Apple gets mostly away with complete breakage every few releases since they have a considerable cult following that is happy to pay for anything they do even if they moan in the GOG and Steam forums later when things don't work anymore. And there is an ever increasing number of (indie) developers that has stopped supporting macOS since it is a constant hassle.
TL;DR: IMO doing such changes just to “make things more elegant” will destroy the Linux Desktop irreparably. And there will only be a decreasing number of users left that only use open source software and wine/proton stuff.
Note: I do not care about EAC breakage (except on principle) since I refuse to play games that use it, but I'm sure that Shovel Knight is not the only native release that is affected by this change.
Apple gets mostly away with complete breakage every few releases since they have a considerable cult following that is happy to pay for anything they do even if they moan in the GOG and Steam forums later when things don't work anymore. And there is an ever increasing number of (indie) developers that has stopped supporting macOS since it is a constant hassle.
TL;DR: IMO doing such changes just to “make things more elegant” will destroy the Linux Desktop irreparably. And there will only be a decreasing number of users left that only use open source software and wine/proton stuff.
10 Likes, Who?
Quoting: GuestQuoting: ShabbyXThis is absolutely not true. 16KB is 4 pages of memory, saving that on every .so is huge! It's not just that you have the memroy laying around, there are other costs too. There's the cost of loading the objects from disk, maintaing the struct page entries in the kernel etc.Everything has worked fine so far even with this extra cost, so I doubt the (real world) effect is huge. What kind of improvement this change makes for desktop use case?
Quoting: ShabbyXThere is a reason Linux is _fast_. With your approach, Linux would have been bloatware like the rest of them.How come Linux is the fastest kernel there is when it absolutely follows that "bloatware" practice?
1. As others mentioned, little things add up. Linux is fast because every little performance improvement is applied. After all, large company X saves a lot of money for improving things by 0.01% simply because the multpilier is so large for them. You enjoy a fast kernel on desktop thanks to that.
2. Two reasons. One is that a good chunk of the ABI people use is POSIX, which is standardized. Linux is not free to change it, no matter how many complaints they may have about it.
But more importantly, it's because Linux actually doesn't follow the bloatware practice. Linux's ABI most definitely changes in backwards incompatible ways. It just happens to change mostly in actively developed areas where users are also developers of the feature and they adapt to new changes.
Linux's motto is not *never change the ABI*, but *never break userspace*. The difference is that if a change breaks ABI but not userspace (like, no active users of it, or userspace happens to not break), then the change goes through perfectly fine.
---
To be clear, I'm not defending glibc. They were wrong to make a backwards incompatible change without incrementing the major version. I'm only saying that "win32 is stable, so it must be good" is a terrible argument.
Last edited by ShabbyX on 17 August 2022 at 1:28 pm UTC
3 Likes, Who?
Patreon
PayPal
See more from me