Recent NVIDIA drivers are causing issues for Proton

Have you been seeing some problems with the NVIDIA 535 driver series when trying to run Steam games on Linux with Proton? Well, you're not alone but investigations are under way. Here's what's happening.

Back in June there was a report on the Proton GitHub where a user noticed how Proton 8 and Proton Experimental were unable to create a new prefix. A prefix being the special directory set up by Proton, where it stores all the expected Windows files each game needs to run. Quite a few comments have confirmed it, including developers working on Proton. 

The issue is an odd one, because even if you're on an AMD GPU and happen to have some NVIDIA driver bits installed (say from moving from an NVIDIA GPU to an AMD GPU), the issue can still happen. The set up gets stalled on DXSETUP and just won't go any further.

On the GitHub issue report a Proton developer just recently mentioned this:

Good news - there's some progress. The issue seems to be due to address space exhaustion. DXSETUP is a 32 bit process and it can address 4G of memory. Nvidia's driver starting with 535 seem to be a bit more heavy than the previous series. While the bump is not colossal it's enough to make things crash. Nvidia is aware of this.

One thing that should help with the issue is uninstalling all the other Vulkan drivers that are not in use. So if you are on Nvidia uninstall all things Mesa, including extra Vulkan layers, and vice versa. Otherwise they are all loaded using the precious address space.

That said I'm looking into where the memory goes in places we have control over it and why DXSETUP is this pathological.

It seems like if you have the issue you may be able to use Proton 7 initially to try to run a game (even if it won't work in Proton 7) and then swap over to Proton 8 or Proton Experimental.