Valve dev understandably not happy about glibc breaking Easy Anti-Cheat on Linux

QuoteWe are very interested in helping with any underlying resource constraints.

i understand that those projects need funding, but didnt that create an incentive to create problems so they can charge for an solution? not only on that project, but in others that listen that there is money to be made on breaking stuff...

QuoteMeanwhile, you can still write and run winmm[1] applications on Windows 10, and they will work in almost all cases. It's simply the case that the win32 API is more stable than Linux userland APIs, so it's entirely plausible that games will run better in Wine, which shares that stable ABI, than they will on Linux, especially as time goes on and Linux userland shifts yet again.

i said that a loong time ago, and people insisted that i was wrong.
hell even linus torvalds complained about this.

i wonder how the world would look like if we didnt had windows and as result wine, or how it will look like if windows die in the future, and if it didnt had consoles, retro gaming would be history...

Quoting: Beamboom

Quoting: Eike

Quoting: Beamboom

Quoting: 1xokThis whole criticism from CodeWeavers developers should be taken with a pinch of salt...

... Why?

Because WINE is where their money comes from?
It's not like they're a neutral party in this question...

I see nothing wrong or biased in what they said.

New releases of WINE/Proton breaks games that worked with the older version from time to time so yes they are wrong in that WIN32 is the only stable ABI on Linux since it clearly isn't stable. There is a reason why Steam still have several versions of Proton since 3.7 available to choose from.

Quoting: landeelglibc compatibility has been a pain for a while.
I have released 2 games on Steam with native Linux ports.
I'm always afraid of doing a dist-upgrade, because it's very likely the compiled binaries won't run on the previous distro version.
It's absurd, but targeting win32 is more stable.
There should be a way to have multiple working glibc versions.

There are, you can also in your code tell a new version of GLIBC that you want the function from the older version X since GLIBC uses versioned symbols and keep the old ones around since they started to do that. It's a very painful process though since one have to do this for every single symbol one want's to use.

Quoting: Klaas

Quoting: Purple Library GuyUm, how many of these other thingies are we typically talking about existing at a time?

Run ldd on a binary and count the number of libraries that are referenced.

No. I won't RTFM either.
In any case, would that tell me much about the total system usage?


Last edited by Purple Library Guy on 17 August 2022 at 6:11 pm UTC

Quoting: minidou

Quoting: Guest

Quoting: minidouI fail to see how glibc devs could be blamed for removing a function deprecated for 17 years (though I can agree a major version bump should have been called for). Do videogame devs need that much nursing ? EAC linux lib is a recent piece of software, how many deprecated dependencies do they use ?

The blame is not really about removing the deprecated feature. Blame is for the failure to restore it after discovering that users for the feature still exist and the removal broke stuff.

They did restore it because of the backlash, but they really shouldn't have, there's no reason for it. This isn't some obscure lib we're talking about but the GNU C library. If a handful of program gets broken it's on them, the function was marked deprecated almost two decades ago. An outstanding majority of program have no issue at all with the removal of this function, why should a niche but maybe high visibility user dictate the direction of glibc ?

Something not discussed here is that the removal of DT_HASH allows a save of about 1% or 16kB of space per Glibc shared object. This is an improvement. There clearly is a good reason for finally removing this deprecated function. But we shouldn't profit from this improvement because of some devs bad practice ?

Quoting: TheSHEEEP

Quoting: GuestThe blame is not really about removing the deprecated feature. Blame is for the failure to restore it after discovering that users for the feature still exist and the removal broke stuff.

Yes, a lot of developers (especially newer, younger ones that lack the experience) argue in favor of always using the very latest everything and always keeping everything updated.
I've seen that time and time again in several teams.

But that's not a position that can be maintained in reality.
You'd require armies just to maintain old stuff.

It's fine to remove a function if you can't see that anyone is using it. An understandable mistake.

This is off topic. The impacted softwares were initialised years after DT_HASH was marked deprecated. In the case of EAC lib, probably a good 15 years later. In every sector of the software industry this would be considered bad practice, but VG softwares get a pass for some reason.

Quoting: TheSHEEEPBut the moment you realize that there were indeed many still using it, it should clearly be restored (with a deprecation marker, but still).

They did that, almost two decades ago.

And it seems there isn't that many users impacted. I can count the EAC lib and some lib used in shovel night. Compared to the millions of programs using glibc.

I'm also wondering if this opens a hole in EAC. Aka since they only seems to check DT_HASH then I can craft a .so that have a nice DT_HASH for EAC to be happy about and then I create my real stuff in the DT_GNU_HASH section that is what the system really loads instead. Now I have never played around with low level linking like this so I don't know if this is possible, but it sure sounds like it.

On a slightly tangential note, this is a long but very interesting article about the sh*t-show that is ABI-compatibility in C(++)-land (reminder: the languages of choice for many games). There are a couple other similar articles on that blog (1, 2, also very enjoyable if you have the time to spare.

Quoting: JpxeAlso isn't this the the same on iOS, Android and macOS. Aren't they always changing things and "breaking" backwards compatibility? They still seem to be a very viable target for third-party developers.

Apple products also suck balls. Criticism of Linux is still valid.
With that said, what prevents all games from shipping old system libraries?

Quoting: Purple Library GuyIn any case, would that tell me much about the total system usage?

Only if you do that for everything that you use (ignoring duplicates). It's very hard to estimate as it hugely varies between programs and obviously the number of programs that you use simultaneously.

The upper bound is the number of unique files in /usr/lib, although that should overshoot the real number by a huge amount.

Note: I did not ralead through the 70+ comments, but...

QuoteSituations like this can be pretty messy and this is not a case of open source versus secret closed source anti-cheat stuff either, since the glibc issue affected a Native Linux game (Shovel Knight) and Linux software libstrangle. No doubt there are other things yet to be discovered that were broken by the change.

Not entirely accurate as if the games themselves were also open source, they can be modified and compiled with fixes to deprecated stuff and use new features. This is why Stallman has always said that games would be better to use open source engines, and could have proprietary data.

I will have to read up on the rest of the thoughts, and why some function was removed / changed in libc.

Also, while Wine is fantastic at getting windows of all versions to work under Linux, Windows itself may still 'support' some old stuff, but that doesn't mean some of the implementations still work in Windows 10/11.

Look at all the SafeDisk stuff that will no longer work without dubious cracks.

LSB was a good idea. We need more automated testing to catch that kind of bugs. FULLY cover important APIs.

If an application requires qn older "base" or an older libraries, shims should be provided, and that should be exercised with tests.

Of course, that requires a substantial ammount of effort :/

This is why we in the "enterprise" world of servers prefer distros such as RHEL which backport security and bugfix patches but retain compatibility with existing software for a number of years by locking software versions instead of a rolling release distro like Arch with constant new versions. Yet we were always judged for running "old software" by the general non-admin linux crowd 😂

Maybe now they can understand why we do it that way. 😂

With that said, we shouldn't stop developing or pushing ahead in our FOSS software by holding back development just to keep some proprietary spyware like EAC working that doesn't play nice in the first place.

Backporting security fixes can lead to a lot of trouble if something is missed. Remember the Debian OpenSSL bug a few years ago?

I was speaking more along the lines of RHEL or SLES.

For example, RH very much keep up with latest CVEs and catalogue them for their customers and detail whether it's fixed, unaffected or vulnerable:

https://access.redhat.com/security/security-updates/#/cve

With that said having the latest versions of software doesn't make you immune from security issues either, nor less likely.

It can also go the other way, a security issue can occur only in a new version of software but not affect older versions, there have been several security issues over the years that simply didn't affect RHEL because the security bug was only present in newer versions of software.

Also, a security issue can go unnoticed for years in software, even eith latest versions. Then there's developers who deem certain things "not a priority" who don't patch a security bug they know exists (this affects proprietary software like windows mostly)

Don't assume that just because patches are backported that the software is more vulnerable as a result. RH has a lot of professional employed security experts keeping tabs on this.

Edit: Typos.. I hate typing on smartphones...


Last edited by BlackBloodRum on 17 August 2022 at 9:19 pm UTC

Quoting: F.UltraNew releases of WINE/Proton breaks games that worked with the older version from time to time so yes they are wrong in that WIN32 is the only stable ABI on Linux since it clearly isn't stable.

It's not a either/or, binary reality discussed here. It's not a question if this or that ever breaks at all, ever.

Yes, obviously. But my point was that the supposedly stable things have large caveats as well and it is really not related to the current issue at all to constantly state that rolling release distribution break everything constantly. This is not an issue of rolling vs frozen distribution.

I don't think things like this breakage can be tested automatically since you'd have to create so many obscure test cases that the effort would make it infeasible.

Quoting: KlaasYes, obviously. But my point was that the supposedly stable things have large caveats as well and it is really not related to the current issue at all to constantly state that rolling release distribution break everything constantly. This is not an issue of rolling vs frozen distribution.

I don't think things like this breakage can be tested automatically since you'd have to create so many obscure test cases that the effort would make it infeasible.

It's all on usage case. For a server, stable is better.

Both options have caveats and issues to deal with.

My point was that people (such as yourself) often believed that enterprise level distros were pointless even for servers (You have no idea how many times I've had this exact discussion over the years) and proceed to attack enterprise grade distros, citing false security concerns and caveats and would suggest that running arch on a server was a better option.

Meanwhile in the enterprise world we always cited compatibility as a primary concern which this EAC issue highlights.

So my point was merely it would help you understand why we use such software in a stable environment that allows us to keep our systems running without breaking software.

That rolling isn't always a better option, software changes and it's the nature of the beast, it's how you handle those changes that matters.

In my case I use both, I use rolling release for my desktop and stable for servers. Best of both worlds 🫠


Last edited by BlackBloodRum on 17 August 2022 at 9:30 pm UTC

I am not a software developer or a progammer; I am just a regular computer user. So please do not get angry at me for asking this, as I probably don't know what I'm talking about.

But wouldn't running games, and programs/applications in general, inside containers fix this kind of issue? Isn't that one of the goals of Flatpak?


Last edited by CyborgZeta on 17 August 2022 at 10:12 pm UTC

I wasn't going to comment on this because it is way off-topic, but having thought about it for a while I decided to do so anyway, in the interest of public education on neural diversity. Liam or moderators may remove this if they think it inappropriate.

Quoting: dibzYeah, one of the best and worst things about Linux is the vocal community. And let's be honest, the people that flock tend to have very strong personalities -- borderline autistic at times.

Autistic is not the word you are looking for here. It is a common and widespread misconception, but it really isn't what you are looking for to describe the individuals you are hinting at. Autistics are often asocial, generally not antisocial. Typically, if an autistic person learns they have said or done something that have hurt someone, they will feel intense remorse.

A "borderline autistic" person is a perfectly average person, you likely won't even notice anything different except perhaps slightly increased anxiety compared to baseline human their age. At most you might describe them as someone who likes to stay home and read a book on a Saturday every now and then.

They are not people who will try to rally the masses, they will probably just stand at the sideline wishing they could contribute more. There are exceptions though, a certain fellow countrywoman is a strong personality who does quite a bit to hold people accountable for the climate.

Quoting wikipedia,

QuoteThe autism spectrum is a range of neurodevelopmental conditions generally characterized by difficulties in social interactions and communication, repetitive behaviours, intense interests, and unusual responses to sensory stimuli.

The difficulty in social interaction and communication refers to discomfort and avoidance and reduced body language. But many autistics won't put themselves in most social situations to begin with, either because of a lack of interest in social activities or because it confuses them. Even mailing-lists, or forum posting is something that scares many.

Source: I've done a lot of reading on autism, I have talked to a lot of professionals, I have friends who are autistic and I am autistic myself. I am however far from an expert, and open to learn more.

PS. What I have described as autistics above is not a set of all people with autism spectrum disorder, it is a generalised set of the ones who are high-functioning enough that you are likely to have been in contact with them in something as complex as technical online discussions.

Quoting: CyborgZetaI am not a software developer or a progammer; I am just a regular computer user. So please do not get angry at me for asking this, as I probably don't know what I'm talking about.

But wouldn't running games, and programs/applications in general, inside containers fix this kind of issue? Isn't that one of the goals of Flatpak?

Yes and no.

It's not so much a technical limitation at this point, but rather a legal one.

It could be done in a number of ways but EAC, being proprietary, cannot bundle a glibc binary into their closed source product for redistribution in another closed source product (the game) for legal reasons since it breaks to LGPL terms.

In addition the glibc devs heavily advise to always dynamically link to the OS's glibc for best overall compatibility and security.

(EAC did correctly here by dynamically linking)

Usually this isn't a big deal in FOSS projects because someone will come along and patch an application to work in a newer glibc and happy days.

But the kicker here is that EAC is compiled directly into the distributed games. Which means distro vendors can't patch it, steam can't patch it and neither can EAC themselves.

The only way to properly fix it (to work with newer glibc) at this point in deployed EACs would be for EAC to update EAC, and then each EAC game updates their EAC and recompiles with the patch.

The other legal issue is that neither EAC nor steam can just come along and just stick a closed source game into a new flatpak with needed libraries, because that would break the games license terms.

Considering many games may not see an update due to inactive devs, it doesn't look good.

At this point best case is that EAC patch to use the modern hash and the game devs update their EAC.

PS: Sorry if I came off a bit brash earlier.. long and bad day...

Quoting: CyborgZetaI am not a software developer or a progammer; I am just a regular computer user. So please do not get angry at me for asking this, as I probably don't know what I'm talking about.

But wouldn't running games, and programs/applications in general, inside containers fix this kind of issue? Isn't that one of the goals of Flatpak?

A great question CyborgZeta!

Yes and no. It is really quite overkill to use something like flatpak for the use cases we are talking about, the industry have dealt with this issue for decades and the solution (which is typically used on the windows platform) is to bundle your dependencies either statically in your binary or as dynamic libraries next to your binary.

There are pros and cons to that, for something like a game it is usually "good enough" to bundle only some dependencies given the non-critical nature of games. Programs written in Go(lang) are built as independent static binaries by design. As far as I know this means that they will run pretty much on any system as long as the architecture is compatible (linux x86, windows x86, linux arm, etc).

For drawbacks there are three big ones, one is size. Statically built binaries are larger in size since they bundle everything the program needs inside the binary itself. Another is that if the assumptions those dependencies made about the platform were to change they would still break. Lastly, security. If my program uses curl dynamically and a security flaw is discovered in curl that can be updated independently of my program.

Containerised solutions share the same first two drawbacks and has the third built in "by design" (both good and bad).

Another one you can make a solid argument for is that containerised applications like flatpak add a lot of unnecessary complexity to an already complex chain.

An advantage that a statically linked binary have over a container is reduced resource consumption and a smaller size on disk. A container needs the entire runtime to be present, which will include a ton of stuff the program won't need. Granted, if you install enough containerised applications that rely on the same runtime the differences go away.