Support us on Patreon
PayPalValve has enabled the next step towards making Steam games on Linux run smoother in the latest Steam Beta release.
This is something Valve has been working towards for some time now, as the Steam Client has been able to download pre-compiled GPU shaders, which you might have seen when something pops up in your Steam Downloads with an OpenGL and Vulkan icon below.
In the latest Steam Beta from May 25, it wasn't actually mentioned but there's a new option you can enable in the Shader Pre-Caching settings to "Allow background processing of Vulkan shaders". So it will process in the background and then when you go to click play, if it's not done on that particular title it should then kick into full-gear and attempt to process before loading the game. For those interested in a little background, it's using the Fossilize library and Vulkan layer which you can find on Valve's GitHub.
Note: Right now on NVIDIA the it seems the main processing will only use 1 core due to an issue, hopefully that will be solved soon. On AMD it's able to use multiple threads when it needs to. It appears you can tweak background thread count by going to "steam://open/console" and tweaking the "unShaderBackgroundProcessingThreads" var, but do so at your own risk.
The result should be that you see much improved performance. We're not talking a boost to the maximum framerate but reducing overall stutter. Something that has been a big problem in some games. As the idea is that instead of the game building it all up as it's playing, it's got it all ready for you when you hit play and this applies to both native/supported titles and Proton.
It doesn't just do it for installed games, it will do it as you're downloading them too, so by the time you've finished downloading it might even be all ready.
Article taken from GamingOnLinux.com.
Some you may have missed, popular articles from the last month:
The comments on this article are closed.
So if i get it right, it should reduce shader caching issues right? I'll be testing it for sure now since I have a bit of a problem with that in some games.
3 Likes, Who?
This is a very interesting tech! I wonder how they implemented it. The D3D shaders are stored in the game resources which can be in whatever proprietary format the developers choose so they can't be automatically extracted and transpiled to SPIR-V (the Vulkan binary shader format). Fortunately, this format is standardized at the spec level so you can use the same shader cache on any GPU, be it AMD, NVIDIA or Intel and you won't need to rebuild it each time you update the driver (remember that pain with OpenGL?). It's not a problem for correctly ported games as the port developer (like Feral) can create a native SPIR-V cache ahead of time so the game just loads the shader it needs directly. That's why Shadow of the Tomb Raider runs so smooth natively and stutters like hell on DXVK.
I also had severe freezes in Halo 2 at the level start and often when you get to a new area, it's not just stutter but a full freeze for 2-3 seconds. That's how bad it is. Interestingly, Steam downloaded the shader updates several times even though I didn't play the game at the time! If I played it I could explain it as a shader cache "corruption" from the Steam point of view, like I compiled some more shaders and it found differencies and updated it. But that wasn't the case. And the file was bigger and bigger every time so something definitely is going on behind the scenes.
Still, the only way I see to eliminate this stuttering that's caused by shader transpiling is to actually play the game so it loads the shaders and causes DXVK to process and cache them. So how do Valve do that? Do they have a dedicated employee that plays non-native games from start to finish to cause it to load all the shaders it has? I doubt that's the case. If you look at the %STEAMROOT%/steamapps/shadercache/%appid%/ directory it's pretty interesting. Let's take a look at Halo (976730):
Ok, the usual DXVK cache is there but it's really small, only 3.1 Mb. steamapp_pipeline_cache.foz is 51 Mb, steam_pipeline_cache.foz is 285 Mb. Probably that's the main one. GLCache is smaller (64 Mb in total, probably the same stuff you can find in your ~/.nv directory; weird, I have no idea why there's even any mention of OpenGL when all I use is DXVK) but the suspicious directory is swarm (313 Mb, the individual files are 287 Mb and 26 Mb). That's a p2p-like name and I suspect Steam uploads players caches to their servers and combines them into a shared cache to push it back to everybody later. That's how they might crowdsource the shaders.
I couldn't find any information regarding this system and it would be very interesting to read how it all works.
Edit: oh well, it says it right there, lol:
Last edited by rkfg on 26 May 2020 at 10:22 am UTC
I also had severe freezes in Halo 2 at the level start and often when you get to a new area, it's not just stutter but a full freeze for 2-3 seconds. That's how bad it is. Interestingly, Steam downloaded the shader updates several times even though I didn't play the game at the time! If I played it I could explain it as a shader cache "corruption" from the Steam point of view, like I compiled some more shaders and it found differencies and updated it. But that wasn't the case. And the file was bigger and bigger every time so something definitely is going on behind the scenes.
Still, the only way I see to eliminate this stuttering that's caused by shader transpiling is to actually play the game so it loads the shaders and causes DXVK to process and cache them. So how do Valve do that? Do they have a dedicated employee that plays non-native games from start to finish to cause it to load all the shaders it has? I doubt that's the case. If you look at the %STEAMROOT%/steamapps/shadercache/%appid%/ directory it's pretty interesting. Let's take a look at Halo (976730):
976730
├── DXVK_state_cache
│ └── MCC-Win64-Shipping.dxvk-cache
├── fozpipelinesv4
│ ├── steamapp_pipeline_cache.foz
│ └── steam_pipeline_cache.foz
├── nvidiav1
│ └── GLCache
│ └── ad4b934e19c9451ee3bd46c449bff6c8
│ └── 30d616d94318fb8e
│ ├── steamapp_shader_cache0.bin
│ ├── steamapp_shader_cache0.toc
│ ├── steam_shader_cache.bin
│ └── steam_shader_cache.toc
└── swarm
└── steam_shader_cache
├── bucket_1695.swarc
└── bucket_211696.swarcOk, the usual DXVK cache is there but it's really small, only 3.1 Mb. steamapp_pipeline_cache.foz is 51 Mb, steam_pipeline_cache.foz is 285 Mb. Probably that's the main one. GLCache is smaller (64 Mb in total, probably the same stuff you can find in your ~/.nv directory; weird, I have no idea why there's even any mention of OpenGL when all I use is DXVK) but the suspicious directory is swarm (313 Mb, the individual files are 287 Mb and 26 Mb). That's a p2p-like name and I suspect Steam uploads players caches to their servers and combines them into a shared cache to push it back to everybody later. That's how they might crowdsource the shaders.
I couldn't find any information regarding this system and it would be very interesting to read how it all works.
Edit: oh well, it says it right there, lol:
Last edited by rkfg on 26 May 2020 at 10:22 am UTC
4 Likes, Who?
Quoting: rkfgThis is a very interesting tech! I wonder how they implemented it. The D3D shaders are stored in the game resources which can be in whatever proprietary format the developers choose so they can't be automatically extracted and transpiled to SPIR-V (the Vulkan binary shader format).
Actually, Microsoft open-sourced a compiler to turn HLSL (the DirectX shader language) shaders into SPIR-V, which has been integrated into the Vulkan ecosystem.
QuoteNow, thanks to Microsoft open sourcing their HLSL compiler, the Vulkan community has been able to work cooperatively with Microsoft to make HLSL support in Vulkan a proven choice alongside Khronos’ own GLSL shading language.
...
With feedback from the Vulkan community, Google, LunarG, NVIDIA, and Valve have steered an effort to resolve these and other issues by adding SPIR-V legalization to Vulkan’s HLSL shader compilers.
Edit to add: I have no idea whether Fossilise or DXVK or anything else actually uses DXC, but it's there.
Last edited by CatKiller on 26 May 2020 at 10:42 am UTC
2 Likes, Who?
It only uses one thread on my system (zen3 and polaris [mesa master]).
Last edited by lod on 26 May 2020 at 10:45 am UTC
Last edited by lod on 26 May 2020 at 10:45 am UTC
0 Likes
Excellent news! This stutter in the beginning of a new scene while using Proton is really annoying in some cases. If this is integrated well enough, I'm convinced that this improves the user experience by far. Again, great job Valve (now please also fix hardware accelerated in-home streaming for AMD users ^_^ ).
0 Likes
Quoting: PatolaAny way to follow the shader transpilations that are occuring in the background?
Open up Steam's settings - Shader Caching
It will show an estimate time about finishing the current game it compiles shaders with showing the name of game also.
1 Likes, Who?
Schattenspiegel May 26, 2020
Cool! I like the tech and I love to use it, but if they do not finally fix the annoying re-download of shader cashes every time I start up steam for good, I will not use it ever again! ...probably! ...at least for a day ...or two! :P
1 Likes, Who?
Quoting: rkfgFortunately, this format is standardized at the spec level so you can use the same shader cache on any GPU, be it AMD, NVIDIA or Intel and you won't need to rebuild it each time you update the driver (remember that pain with OpenGL?). It's not a problem for correctly ported games as the port developer (like Feral) can create a native SPIR-V cache ahead of time so the game just loads the shader it needs directly. That's why Shadow of the Tomb Raider runs so smooth natively and stutters like hell on DXVK.You are misunderstanding what SPIR-V is. It's not something that the gpu can execute, it still needs to be combined with more pipeline state information and then optimized and compiled to the specific gpu's machine code. This is very expensive to do and causes the stuttering with DXVK. And that's where fossilize comes into play. It collects all pipeline state and all shaders that are needed to create the pipelines (basically gpu programs) and then Steam distributes them so that you can compile them for your specific driver and gpu outside of the game. That way the driver can cache the pipelines and doesn't need to compile them from scratch while you're playing.
As for why Feral ports don't stutter like DXVK, D3D11 doesn't give enough information on how a shader will be used before actually using it, that's why DXVK can only compile the pipeline when the game draws something with it. Feral know how shaders will be used since they have access to the game, so that's a big advantage.
Quoting: CatKillerThat's close to useless for Proton since games don't give the driver HLSL, they have bytecode in form of DXBC or the newer DXIL. Also translating shaders to SPIR-V is quite fast and not the bottleneck in DXVK afaik, the problem is the missing state information so that pipelines can only be compiled at draw time without a state cache hit.Quoting: rkfgThis is a very interesting tech! I wonder how they implemented it. The D3D shaders are stored in the game resources which can be in whatever proprietary format the developers choose so they can't be automatically extracted and transpiled to SPIR-V (the Vulkan binary shader format).
Actually, Microsoft open-sourced a compiler to turn HLSL (the DirectX shader language) shaders into SPIR-V, which has been integrated into the Vulkan ecosystem.
Last edited by DadSchoorse on 26 May 2020 at 11:56 am UTC
7 Likes, Who?
Quoting: DadSchoorseThat's close to useless for Proton since games don't give the driver HLSL, they have bytecode in form of DXBC or the newer DXIL. Also translating shaders to SPIR-V is quite fast and not the bottleneck in DXVK afaik, the problem is the missing state information so that pipelines can only be compiled at draw time without a state cache hit.
I suspected that was the case, but there's the ability to handle those hopefully coming too. It was the "how do we turn things into SPIR-V?" part that I was commenting on.
While Microsoft has historically used many methods to lock people into their ecosystem, shaders don't seem to be one that they're trying: developers with their existing DirectX knowledge and existing DirectX tooling can relatively easily produce things for Vulkan.
1 Likes, Who?
See more from me