Steam Beta adds Vulkan shader processing

So if i get it right, it should reduce shader caching issues right? I'll be testing it for sure now since I have a bit of a problem with that in some games.

This is a very interesting tech! I wonder how they implemented it. The D3D shaders are stored in the game resources which can be in whatever proprietary format the developers choose so they can't be automatically extracted and transpiled to SPIR-V (the Vulkan binary shader format). Fortunately, this format is standardized at the spec level so you can use the same shader cache on any GPU, be it AMD, NVIDIA or Intel and you won't need to rebuild it each time you update the driver (remember that pain with OpenGL?). It's not a problem for correctly ported games as the port developer (like Feral) can create a native SPIR-V cache ahead of time so the game just loads the shader it needs directly. That's why Shadow of the Tomb Raider runs so smooth natively and stutters like hell on DXVK.

I also had severe freezes in Halo 2 at the level start and often when you get to a new area, it's not just stutter but a full freeze for 2-3 seconds. That's how bad it is. Interestingly, Steam downloaded the shader updates several times even though I didn't play the game at the time! If I played it I could explain it as a shader cache "corruption" from the Steam point of view, like I compiled some more shaders and it found differencies and updated it. But that wasn't the case. And the file was bigger and bigger every time so something definitely is going on behind the scenes.

Still, the only way I see to eliminate this stuttering that's caused by shader transpiling is to actually play the game so it loads the shaders and causes DXVK to process and cache them. So how do Valve do that? Do they have a dedicated employee that plays non-native games from start to finish to cause it to load all the shaders it has? I doubt that's the case. If you look at the %STEAMROOT%/steamapps/shadercache/%appid%/ directory it's pretty interesting. Let's take a look at Halo (976730):

 
976730
├── DXVK_state_cache
│   └── MCC-Win64-Shipping.dxvk-cache
├── fozpipelinesv4
│   ├── steamapp_pipeline_cache.foz
│   └── steam_pipeline_cache.foz
├── nvidiav1
│   └── GLCache
│       └── ad4b934e19c9451ee3bd46c449bff6c8
│           └── 30d616d94318fb8e
│               ├── steamapp_shader_cache0.bin
│               ├── steamapp_shader_cache0.toc
│               ├── steam_shader_cache.bin
│               └── steam_shader_cache.toc
└── swarm
    └── steam_shader_cache
        ├── bucket_1695.swarc
        └── bucket_211696.swarc
Ok, the usual DXVK cache is there but it's really small, only 3.1 Mb. steamapp_pipeline_cache.foz is 51 Mb, steam_pipeline_cache.foz is 285 Mb. Probably that's the main one. GLCache is smaller (64 Mb in total, probably the same stuff you can find in your ~/.nv directory; weird, I have no idea why there's even any mention of OpenGL when all I use is DXVK) but the suspicious directory is swarm (313 Mb, the individual files are 287 Mb and 26 Mb). That's a p2p-like name and I suspect Steam uploads players caches to their servers and combines them into a shared cache to push it back to everybody later. That's how they might crowdsource the shaders.

I couldn't find any information regarding this system and it would be very interesting to read how it all works.

Edit: oh well, it says it right there, lol: !link


Last edited by rkfg on 26 May 2020 at 10:22 am UTC

This is a very interesting tech! I wonder how they implemented it. The D3D shaders are stored in the game resources which can be in whatever proprietary format the developers choose so they can't be automatically extracted and transpiled to SPIR-V (the Vulkan binary shader format).

Actually, Microsoft open-sourced a compiler to turn HLSL (the DirectX shader language) shaders into SPIR-V, which has been integrated into the Vulkan ecosystem.

Now, thanks to Microsoft open sourcing their HLSL compiler, the Vulkan community has been able to work cooperatively with Microsoft to make HLSL support in Vulkan a proven choice alongside Khronos’ own GLSL shading language.
...
With feedback from the Vulkan community, Google, LunarG, NVIDIA, and Valve have steered an effort to resolve these and other issues by adding SPIR-V legalization to Vulkan’s HLSL shader compilers.

Edit to add: I have no idea whether Fossilise or DXVK or anything else actually uses DXC, but it's there.


Last edited by CatKiller on 26 May 2020 at 10:42 am UTC

It only uses one thread on my system (zen3 and polaris [mesa master]).


Last edited by lod on 26 May 2020 at 10:45 am UTC

Excellent news! This stutter in the beginning of a new scene while using Proton is really annoying in some cases. If this is integrated well enough, I'm convinced that this improves the user experience by far. Again, great job Valve (now please also fix hardware accelerated in-home streaming for AMD users ^_^ ).

Any way to follow the shader transpilations that are occuring in the background?

Open up Steam's settings - Shader Caching

It will show an estimate time about finishing the current game it compiles shaders with showing the name of game also.

Cool! I like the tech and I love to use it, but if they do not finally fix the annoying re-download of shader cashes every time I start up steam for good, I will not use it ever again! ...probably! ...at least for a day ...or two! :P

Fortunately, this format is standardized at the spec level so you can use the same shader cache on any GPU, be it AMD, NVIDIA or Intel and you won't need to rebuild it each time you update the driver (remember that pain with OpenGL?). It's not a problem for correctly ported games as the port developer (like Feral) can create a native SPIR-V cache ahead of time so the game just loads the shader it needs directly. That's why Shadow of the Tomb Raider runs so smooth natively and stutters like hell on DXVK.

You are misunderstanding what SPIR-V is. It's not something that the gpu can execute, it still needs to be combined with more pipeline state information and then optimized and compiled to the specific gpu's machine code. This is very expensive to do and causes the stuttering with DXVK. And that's where fossilize comes into play. It collects all pipeline state and all shaders that are needed to create the pipelines (basically gpu programs) and then Steam distributes them so that you can compile them for your specific driver and gpu outside of the game. That way the driver can cache the pipelines and doesn't need to compile them from scratch while you're playing.
As for why Feral ports don't stutter like DXVK, D3D11 doesn't give enough information on how a shader will be used before actually using it, that's why DXVK can only compile the pipeline when the game draws something with it. Feral know how shaders will be used since they have access to the game, so that's a big advantage.

This is a very interesting tech! I wonder how they implemented it. The D3D shaders are stored in the game resources which can be in whatever proprietary format the developers choose so they can't be automatically extracted and transpiled to SPIR-V (the Vulkan binary shader format).

Actually, Microsoft open-sourced a compiler to turn HLSL (the DirectX shader language) shaders into SPIR-V, which has been integrated into the Vulkan ecosystem.

That's close to useless for Proton since games don't give the driver HLSL, they have bytecode in form of DXBC or the newer DXIL. Also translating shaders to SPIR-V is quite fast and not the bottleneck in DXVK afaik, the problem is the missing state information so that pipelines can only be compiled at draw time without a state cache hit.


Last edited by DadSchoorse on 26 May 2020 at 11:56 am UTC

That's close to useless for Proton since games don't give the driver HLSL, they have bytecode in form of DXBC or the newer DXIL. Also translating shaders to SPIR-V is quite fast and not the bottleneck in DXVK afaik, the problem is the missing state information so that pipelines can only be compiled at draw time without a state cache hit.

I suspected that was the case, but there's the ability to handle those hopefully coming too. It was the "how do we turn things into SPIR-V?" part that I was commenting on.

While Microsoft has historically used many methods to lock people into their ecosystem, shaders don't seem to be one that they're trying: developers with their existing DirectX knowledge and existing DirectX tooling can relatively easily produce things for Vulkan.

https://github.com/rcpoison/dxvk-cache-pool

Fortunately, this format is standardized at the spec level so you can use the same shader cache on any GPU, be it AMD, NVIDIA or Intel and you won't need to rebuild it each time you update the driver (remember that pain with OpenGL?). It's not a problem for correctly ported games as the port developer (like Feral) can create a native SPIR-V cache ahead of time so the game just loads the shader it needs directly. That's why Shadow of the Tomb Raider runs so smooth natively and stutters like hell on DXVK.

You are misunderstanding what SPIR-V is. It's not something that the gpu can execute, it still needs to be combined with more pipeline state information and then optimized and compiled to the specific gpu's machine code. This is very expensive to do and causes the stuttering with DXVK. And that's where fossilize comes into play. It collects all pipeline state and all shaders to create the pipelines (basically gpu programs) and then Steam distributes them so that you can compile them for your specific driver and gpu outside of the game. That way the driver can cache the pipelines and doesn't need to compile them from scratch while you're playing.
As for why Feral ports don't stutter like DXVK, D3D11 doesn't give enough information on how a shader will be used before actually using it, that's why DXVK can only compile the pipeline when the game draws something with it. Feral know how shaders will be used since they have access to the game, so that's a big advantage.

Thanks, that's new to me! I thought SPIR-V is executed directly by the underlying driver since it's in the standard. But yeah, the very low hardware level is still different for every GPU so indeed some sort of translation happens under the hood. I didn't know the bottleneck is in the final compilation for the GPU. Good explanation, I was wondering why we need more preprocessing with this new feature if Steam already provides the cache. Now it's more clear.

Also, in the Fossilize README there's this:

The goal for this project is to cover some main use cases:
<...>
Serialize state in application once, replay on N devices to build up VkPipelineCache objects without having to run application.

I think this is exactly what happens because when Steam is doing this new task there's a fossilize_replay process mentioned in nvidia-smi output. Would be really interesting to see how smooth it all is now out of the box, will try with a good-looking F2P game later.


Last edited by rkfg on 26 May 2020 at 11:43 am UTC

Any way to follow the shader transpilations that are occuring in the background?

Open up Steam's settings - Shader Caching

It will show an estimate time about finishing the current game it compiles shaders with showing the name of game also.

Wow.
I just bought a new 8 TB disk and then moved my games logical volume to that disk and resized it from 3 to 6 TB. Then I installed basically every game I have on Steam (~620). Got only a few hundred GB left.
Judging from the pace I am seeing, it will take a few days to finish... Will it do that to every installed game?

It depends on how many shaders those games have+ if those actually have Foz pipelines cached on Steam + if you have a NV gpu or AMD

At the moment , for AMD if you click on "run the game " it will go into full blast mode which means it will do that building task multi threaded. Still single threaded on NV but hopefully should be fixed at the upcoming weeks.

I turned off shader pre-caching so long ago because whatever benefits it was supposed to impart were offset by having it interfere, waiting for downloads just before playing a game (or something like that ... it's been so long I can't even remember what the problem was exactly) :P In fact I don't even remember what shader pre-caching was even for --- lol. Nevertheless, it sounds like this is a nice thing.


Last edited by Nanobang on 26 May 2020 at 12:38 pm UTC

Does it work for Windows too ? Or there is already something similar ? Just curious...

Great feature. Thanks Valve!

I noticed this issue on Crucible. I really want to see how it will work on my XR 5700 XT.

I'll be interested to see if people notice differences.

I'll be interested to see if people notice differences.

I saw a huge improvement on Quake Champions which is nearly bad as Warframe and Path of Exile about shaders.

It was smooth af.Previously, it wasn't the case , lots of stutters that makes the game literally unplayable.

When they first introduced this shader caching/sharing thing Steam started to download hundreds of megabytes of shaders… not a very slight increase in disk and bandwidth usage, no. So I disabled it right away…

If you prefer stutter and freezes every time something new appears on screen then yeah. But I don't think it's worth it. If you play games with a lot of shaders (= AAA games) you'd better invest into a bigger and faster SSD because the rest of your hardware should already be quite good. Honestly, bad first time experience isn't worth the saved disk space.

If you prefer stutter and freezes every time something new appears on screen then yeah. But I don't think it's worth it. If you play games with a lot of shaders (= AAA games) you'd better invest into a bigger and faster SSD because the rest of your hardware should already be quite good. Honestly, bad first time experience isn't worth the saved disk space.

No, I don’t have that problem. And I have 5TB of SSD for games.

It depends on the game and mostly applies to non-native ones. If you don't play them probably you're fine. But some games stutter much more than the others and if you have plenty of space why not download those caches? Also you will help other players by uploading your cache back. It's like torrents but 100% legal!