A new Steam Beta is up with Vulkan pipeline dumping and collection along with Steam Play improvements

That "zero-byte update" (if referring to updates taking years) problem has been driving me crazy for weeks. Strangely enough, it wasn't affecting only my Proton games, but my Linux native games as well. DoTA would not update unless I continuously changed download regions and even then, I still would achieve download speeds in KB/s (125 Kb/s max). Hope this resolves that issue.

Also hoping the nVidia 418 driver gets added to the PPA soon so I can test whether or not Monster Hunter is really fixed on it. An Arch user has had luck with it and recommended I check it out. We'll see.

That "zero-byte update" (if referring to updates taking years)

Nope, it is when an update is queued up for Steam Play titles that were of size 0 bytes.

I know there are knowledgeable people here. What the heck is "Vulkan pipeline dumping"? It sounds like either a good thing or something oil companies do when something has gone very wrong.

OMG, you don't know what Vulkan pipeline dumping is??!?!?? You must be so embarrassed.


Last edited by shawnsterp on 7 Mar 2019 at 3:18 am UTC

I know there are knowledgeable people here. What the heck is "Vulkan pipeline dumping"? It sounds like either a good thing or something oil companies do when something has gone very wrong.

Vulkan pipeline dumping is like the DXVK state cache, but it works for any Vulkan application. So native and DXVK through Proton. Like the OpenGL cache in Steam it's also shared*, which means over time we won't have any stutter in any Vulkan (especially Proton+DXVK) game, due to shader compiling, anymore.

* When participating in the Steam Shader Pre-Caching, you will download existing shader caches from the SteamSwarm shader server. After you exit the game, it will upload new generated shaders to the server.

I know there are knowledgeable people here. What the heck is "Vulkan pipeline dumping"? It sounds like either a good thing or something oil companies do when something has gone very wrong.

Vulkan pipeline dumping is like the DXVK state cache, but it works for any Vulkan application. So native and DXVK through Proton. Like the OpenGL cache in Steam it's also shared*, which means over time we won't have any stutter in any Vulkan (especially Proton+DXVK) game, due to shader compiling, anymore.

So for Proton games, does it work on top of the DXVK state cache ? Does that mean we now have three levels of caching (pipeline, state, shader) ?

So for Proton games, does it work on top of the DXVK state cache ? Does that mean we now have three levels of caching (pipeline, state, shader) ?

I've checked that with one game (Sniper Ghost Warrior 2), renamed the DXVK_state_cache folder and started the game with Proton. The folder was not recreated, so it seems that DXVK state cache is disabled when Vulkan Pipeline Cache is in use. Which makes totally sense, as it is obsolete in Steam Beta now.

I wonder how hard it can possibly be to fix this 0 byte update bug.

I imagine something along the lines...

int updateSize = steamServer.calcUpdateSize();
if (updateSize <= 0) return false;

(Obviously, if it would be that easy, Valve would have solved it in the first attempt.)

OMG, you don't know what Vulkan pipeline dumping is??!?!?? You must be so embarrassed.

Bit harsh don't you think?
Should all Linux gamers know what dumping is?
Aren't the casual "pick up and play" gamers allowed to use Linux and Vulkan without having to learn all this stuff? That's what's so attractive about Windows isn't it? Ease of use an all that, nothing to learn, install and go, right?
I sure as hell didn't buy a PS4 to learn the anatomy of it. Why can't it be the same for Linux?
Patronizing is what turns people away.

Joke...?

I'm just crossing all my fingers and toes that Sekiro is playable through Steam Play at- or shortly after release

I'm just crossing all my fingers and toes that Sekiro is playable through Steam Play at- or shortly after release

I really hope so too, this is the game I've been the most excited about in a long time. It seems they're still using the same engine as Dark Souls 3 and system requirements look similar too, so it's at least pretty likely that it'll work.

I wonder how hard it can possibly be to fix this 0 byte update bug.

I imagine something along the lines...

int updateSize = steamServer.calcUpdateSize();
if (updateSize <= 0) return false;

(Obviously, if it would be that easy, Valve would have solved it in the first attempt.)

Doing a wild guess here but I assume that one major problem is that the checksum of the files on the steam server somehow differs from what the one on the local machine which triggers a download and then it somehow finds out via some other means that all of the files in the package are unchanged so the download size gets to be 0 but that it still needs to do that last step where it's queued up in order for it to change the checksum on the local files.

So doing the simple obvious check that you posted would make the checksum still be wrong on the local side which means that the client will perform needless checks on the steam servers which valve wants to avoid in order to lessen the load (imagine the load generated when several million people launch steam for the day). And this checksum is probably messed up due to proton for some reason.

As I said however just 100% guesswork here from how I would imagine a situation like this arising when building a distribution system like steam.

I know there are knowledgeable people here. What the heck is "Vulkan pipeline dumping"? It sounds like either a good thing or something oil companies do when something has gone very wrong.

I suspect "Vulkan pipeline dumping" is a term invented by someone at Valve, as I don't remember it being in the Vulkan specification.

The Vulkan specification is designed such that many of the Vulkan data structures you create are visible as simple memory blocks in CPU-accessible memory. That makes it easy to save the data structure content to disk. If you will always need the same data structures when you run an application, it may be quicker and more computationally predictable to load their content from disk, rather than re-create them from scratch.

A Vulkan pipeline used by a game ( which may use thousands of pipelines ) would be unlikely to change between one run and the next unless you modify the game's quality settings, so it is a good candidate for saving to disk. So, I would assume the Steam SDK now provides some form of helper function(s) to make cacheing pipelines easier for developers. I may be wrong in that assumption, of course :)


Last edited by etonbears on 7 Mar 2019 at 9:56 pm UTC

I know there are knowledgeable people here. What the heck is "Vulkan pipeline dumping"? It sounds like either a good thing or something oil companies do when something has gone very wrong.

Hello PLG. Out of curiosity, I looked into this some more; the answer requires some explanation of the Vulkan architecture, which may or, more likely, may not be of interest to you. Skip to the last 3 paragraphs if this sort of thing sends you to sleep :)

Vulkan, on Linux, is provided as a shared library for each supported architecture. Your package manager should report "libvulkan1:amd64" and/or "libvulkan1:i386" or similar if you take a look. This does not require that you actually have any valid Vulkan hardware installed. Although this shared library can be provided by anyone, we probably all use a version written by LunarG, sponsored by Valve, which I think was recently donated to Khronos and will likely become the standard desktop version ( Linux, MacOS, Windows ).

This Vulkan library is formally called the "Vulkan Loader", in that it does not provide much function in itself, but coordinates access to Vulkan services provided by "Vulkan ICDs" and "Vulkan Layers".

Vulkan ICDs ( installable client drivers ) provide access to Vulkan-capable hardware through implementing all of the function calls documented in the Vulkan specification, and can additionally advertise extended function calls, usually for new hardware features that are not in the Vulkan specification yet. Vulkan ICDs are VERY lean wrappers around the hardware features, intended for maximum performance at runtime.

Vulkan Layers, on the other hand, are intended to provide all of the additional features that are not normally required at runtime, such as debugging, function call tracing, error reporting, function call parameter checking and validating the Vulkan state ( i.e. checking what you are requesting is complete and sane ).

A Vulkan Layer can implement one or more of the function calls documented in the Vulkan Specification, but unlike ICDs, is not obliged to implement them all. As with ICDs, a layer can offer extended function calls as well.

When an application initializes Vulkan ( a function call in libvulkan1.so ), the loader constructs a call "chain" for each Vulkan function. If any activated layer implements a particular function, that implementation is included in the call chain for the function; the end of the call chain is the guaranteed implementation in the ICD for your hardware.

These call chains are not at all efficient, but are intended to allow all sorts of checking, recording and analysis of an application while it is in development, when the execution speed is of less interest. At runtime, all development layers are deactivated, so the call chain just goes straight to the ICD implementation and the application/game runs as fast as possible.

If you are still awake and reading, you might be asking what this has to do with "Vulkan pipeline dumping". The answer is that Valve have used the Vulkan Layer mechanism to insert their own code into Steam-Launched Vulkan games, without the need to the game developer to do anything at all.

Vulkan on Linux uses 5 possible configuration directories ( to cater for different distributions liking things in different places, I guess )

/usr/local/etc/vulkan/
/usr/local/share/vulkan/
/etc/vulkan/
/usr/share/vulkan/
$HOME/.local/share/vulkan/

The Vulkan loader checks those locations for subdirectories "icd.d", "explicit_layer.d" and "implicit_layer.d", and reads any JSON-format text files in those directories to find which libraries are installed that implement ICDs and Layers.

Note that there are 2 types of Layer. Layers defined inside any "explicit_layer.d" are loaded by an application directly requesting them from code, but Layers defined in any "implicit_layer.d" are loaded automatically depending on environment variables declared in their JSON file.

If you have Steam installed, check $HOME/.local/share/vulkan/ and you should find 4 JSON files declaring implicit Vulkan Layers for 32bit & 64bit versions for steamoverlay and steamfossilize.

While there is no source code available for the libraries declared in these layers, it is clear that they intercept a small number of Vulkan calls to provide the Steam Overlay and Shader/Pipeline caches. If you activate these features in Steam, the correct environment variables will be set to let the Vulkan Loader know it should load those implicit layers.

More interesting, perhaps, is that this provides a non-intrusive mechanism for enhancing any/all Vulkan game presentations. This would allow overlaying data ( as with Steam Overlay ) of your choice from elsewhere, or applying post-process effects to your liking, as with the interesting Windows ENBSeries utility, for example.

Does anyone know if "Vulkan pipeline dumping" is working with the Mesa Radv Driver right now ? I have really bad stuttering in most of my games that are utilizing Proton with DXVK since i have switched to an AMD Radeon RX 590. Games using wined3d and therefore the Mesa GL Driver don't have that problem at all.

Does anyone know if "Vulkan pipeline dumping" is working with the Mesa Radv Driver right now ? I have really bad stuttering in most of my games that are utilizing Proton with DXVK since i have switched to an AMD Radeon RX 590. Games using wined3d and therefore the Mesa GL Driver don't have that problem at all.

Obviously it depends on the game and how efficiently the virtualization provided by Wine and DXVK maps the game to your hardware.

One thing Feral often mention when you start their native games is the effect of the CPU cores not being set to constant performance mode. It takes time for cores to clock down and up again between bursty loads which can lead to stuttering.

You will probably find the instructions for checking and changing the CPU mode on their Web site support FAQ. See if that is affecting you ( I have pkexec scripts to switch into 'game mode' and back to 'stop heating my room' mode, and use them as needed).