Steam Beta adds Vulkan shader processing

So if i get it right, it should reduce shader caching issues right? I'll be testing it for sure now since I have a bit of a problem with that in some games.

This is a very interesting tech! I wonder how they implemented it. The D3D shaders are stored in the game resources which can be in whatever proprietary format the developers choose so they can't be automatically extracted and transpiled to SPIR-V (the Vulkan binary shader format). Fortunately, this format is standardized at the spec level so you can use the same shader cache on any GPU, be it AMD, NVIDIA or Intel and you won't need to rebuild it each time you update the driver (remember that pain with OpenGL?). It's not a problem for correctly ported games as the port developer (like Feral) can create a native SPIR-V cache ahead of time so the game just loads the shader it needs directly. That's why Shadow of the Tomb Raider runs so smooth natively and stutters like hell on DXVK.

I also had severe freezes in Halo 2 at the level start and often when you get to a new area, it's not just stutter but a full freeze for 2-3 seconds. That's how bad it is. Interestingly, Steam downloaded the shader updates several times even though I didn't play the game at the time! If I played it I could explain it as a shader cache "corruption" from the Steam point of view, like I compiled some more shaders and it found differencies and updated it. But that wasn't the case. And the file was bigger and bigger every time so something definitely is going on behind the scenes.

Still, the only way I see to eliminate this stuttering that's caused by shader transpiling is to actually play the game so it loads the shaders and causes DXVK to process and cache them. So how do Valve do that? Do they have a dedicated employee that plays non-native games from start to finish to cause it to load all the shaders it has? I doubt that's the case. If you look at the %STEAMROOT%/steamapps/shadercache/%appid%/ directory it's pretty interesting. Let's take a look at Halo (976730):

 
976730
├── DXVK_state_cache
│   └── MCC-Win64-Shipping.dxvk-cache
├── fozpipelinesv4
│   ├── steamapp_pipeline_cache.foz
│   └── steam_pipeline_cache.foz
├── nvidiav1
│   └── GLCache
│       └── ad4b934e19c9451ee3bd46c449bff6c8
│           └── 30d616d94318fb8e
│               ├── steamapp_shader_cache0.bin
│               ├── steamapp_shader_cache0.toc
│               ├── steam_shader_cache.bin
│               └── steam_shader_cache.toc
└── swarm
    └── steam_shader_cache
        ├── bucket_1695.swarc
        └── bucket_211696.swarc
Ok, the usual DXVK cache is there but it's really small, only 3.1 Mb. steamapp_pipeline_cache.foz is 51 Mb, steam_pipeline_cache.foz is 285 Mb. Probably that's the main one. GLCache is smaller (64 Mb in total, probably the same stuff you can find in your ~/.nv directory; weird, I have no idea why there's even any mention of OpenGL when all I use is DXVK) but the suspicious directory is swarm (313 Mb, the individual files are 287 Mb and 26 Mb). That's a p2p-like name and I suspect Steam uploads players caches to their servers and combines them into a shared cache to push it back to everybody later. That's how they might crowdsource the shaders.

I couldn't find any information regarding this system and it would be very interesting to read how it all works.

Edit: oh well, it says it right there, lol:


Last edited by rkfg on 26 May 2020 at 10:22 am UTC

Quoting: rkfgThis is a very interesting tech! I wonder how they implemented it. The D3D shaders are stored in the game resources which can be in whatever proprietary format the developers choose so they can't be automatically extracted and transpiled to SPIR-V (the Vulkan binary shader format).

Actually, Microsoft open-sourced a compiler to turn HLSL (the DirectX shader language) shaders into SPIR-V, which has been integrated into the Vulkan ecosystem.

QuoteNow, thanks to Microsoft open sourcing their HLSL compiler, the Vulkan community has been able to work cooperatively with Microsoft to make HLSL support in Vulkan a proven choice alongside Khronos’ own GLSL shading language.
...
With feedback from the Vulkan community, Google, LunarG, NVIDIA, and Valve have steered an effort to resolve these and other issues by adding SPIR-V legalization to Vulkan’s HLSL shader compilers.

Edit to add: I have no idea whether Fossilise or DXVK or anything else actually uses DXC, but it's there.


Last edited by CatKiller on 26 May 2020 at 10:42 am UTC

It only uses one thread on my system (zen3 and polaris [mesa master]).


Last edited by lod on 26 May 2020 at 10:45 am UTC

Excellent news! This stutter in the beginning of a new scene while using Proton is really annoying in some cases. If this is integrated well enough, I'm convinced that this improves the user experience by far. Again, great job Valve (now please also fix hardware accelerated in-home streaming for AMD users ^_^ ).

Quoting: PatolaAny way to follow the shader transpilations that are occuring in the background?

Open up Steam's settings - Shader Caching

It will show an estimate time about finishing the current game it compiles shaders with showing the name of game also.

Cool! I like the tech and I love to use it, but if they do not finally fix the annoying re-download of shader cashes every time I start up steam for good, I will not use it ever again! ...probably! ...at least for a day ...or two! :P

Quoting: rkfgFortunately, this format is standardized at the spec level so you can use the same shader cache on any GPU, be it AMD, NVIDIA or Intel and you won't need to rebuild it each time you update the driver (remember that pain with OpenGL?). It's not a problem for correctly ported games as the port developer (like Feral) can create a native SPIR-V cache ahead of time so the game just loads the shader it needs directly. That's why Shadow of the Tomb Raider runs so smooth natively and stutters like hell on DXVK.

You are misunderstanding what SPIR-V is. It's not something that the gpu can execute, it still needs to be combined with more pipeline state information and then optimized and compiled to the specific gpu's machine code. This is very expensive to do and causes the stuttering with DXVK. And that's where fossilize comes into play. It collects all pipeline state and all shaders that are needed to create the pipelines (basically gpu programs) and then Steam distributes them so that you can compile them for your specific driver and gpu outside of the game. That way the driver can cache the pipelines and doesn't need to compile them from scratch while you're playing.
As for why Feral ports don't stutter like DXVK, D3D11 doesn't give enough information on how a shader will be used before actually using it, that's why DXVK can only compile the pipeline when the game draws something with it. Feral know how shaders will be used since they have access to the game, so that's a big advantage.

Quoting: CatKiller

Quoting: rkfgThis is a very interesting tech! I wonder how they implemented it. The D3D shaders are stored in the game resources which can be in whatever proprietary format the developers choose so they can't be automatically extracted and transpiled to SPIR-V (the Vulkan binary shader format).

Actually, Microsoft open-sourced a compiler to turn HLSL (the DirectX shader language) shaders into SPIR-V, which has been integrated into the Vulkan ecosystem.

That's close to useless for Proton since games don't give the driver HLSL, they have bytecode in form of DXBC or the newer DXIL. Also translating shaders to SPIR-V is quite fast and not the bottleneck in DXVK afaik, the problem is the missing state information so that pipelines can only be compiled at draw time without a state cache hit.


Last edited by DadSchoorse on 26 May 2020 at 11:56 am UTC

Quoting: DadSchoorseThat's close to useless for Proton since games don't give the driver HLSL, they have bytecode in form of DXBC or the newer DXIL. Also translating shaders to SPIR-V is quite fast and not the bottleneck in DXVK afaik, the problem is the missing state information so that pipelines can only be compiled at draw time without a state cache hit.

I suspected that was the case, but there's the ability to handle those hopefully coming too. It was the "how do we turn things into SPIR-V?" part that I was commenting on.

While Microsoft has historically used many methods to lock people into their ecosystem, shaders don't seem to be one that they're trying: developers with their existing DirectX knowledge and existing DirectX tooling can relatively easily produce things for Vulkan.

https://github.com/rcpoison/dxvk-cache-pool