Cheese Talks: The Steam Hardware & Software Survey

With Valve's content distribution/social/DRM platform, Steam, having supported all three major desktop platforms for nearly a year now, I've been intending to write something about the Steam Hardware & Software Survey for some time. In the coming months, the impact of Valve's recent SteamOS and Steam Machines announcements is likely to begin shifting some attitudes, perceptions and ultimately (dramatically or subtly) changing the landscape that the survey results depict.

Below is a combination of excerpts and thoughts from the full article (which contains more detail and fancier charts, so take a peek!).



What Is The Survey And Why Should We Care?

From sometime prior to 2004, Valve have been collecting and publicly publishing what is most likely the largest survey of consumer gaming hardware and system configuration ever conducted. What presumably started as a tool for Valve to help understand their audiences has grown into something that gamers, journalists and even other developers get excited about and attempt to draw upon as a resource.

Being tied to a title agnostic distribution platform that spans the breadth of gaming genres, the Steam Hardware & Software Survey is positioned to inherently sidestep most of the issues that face the types hurdles to data collection that technical groups, journalists, individual software developers and hardware vendors encounter when trying to draw understanding of broader industry behaviour. By providing a moderately unobtrusive opt-in prompt to randomly selected users when Steam is launched, the survey is able to target gamers specifically, cross demographic bounds and have a higher likelihood of response.

The frequency and regularity of the survey also provides what is perhaps the most detailed picture of gaming system configuration over time that has ever been compiled.


What Should We Consider When Interpreting The Survey Results?

• Percentages are ratios
• As a distribution platform, Steam has a clear platform bias in title compatibility
• The survey results highlight statistically insignificant fractions
• A best guess margin of error renders anything less than 0.065 percentage points meaningless

How Could The Survey Be Improved?

In the article, I highlight a number of uncertainties and ambiguities that reduce the reliability of the Steam Hardware & Software Survey's results. To finish off, I offered some suggestions to counteract this and boost credibility and confidence.

• Post-publishing changes need to be documented (numbers fluctuate too much after they go live)
• The sampling method needs some level of transparency (this wouldn't impact on the anonymity of the survey, it would just clarify how it's conducted)
• Survey results need more detail on presentation aspects (such as rounding, population size/fluctuation, whether "change" values are percentages or percentage points, what thresholds push things into the "Other" category, and some indication of how invalid or unrecognised values are represented)
• The survey needs a F.A.Q. (see here for what little information I've been able to cobble together over the years)

Whilst looking at platform bias of games available on the Steam store, I discovered that just over 10% of games on Steam offer Linux support, 25.39% of games are playable on a Mac, and all but one title (a Mac specific version of one of the Call of Duty games - thanks to xPaw from SteamDB for quickly identifying that for me!) are Windows compatible.


This lead me to start to look at what the Linux user base percentages would look like if they were adjusted for the number of titles available. By running the percentages found in the survey results across Steam's estimated 54,000,000 active accounts and then dividing those by the number of titles available on each platform, I was able to get an estimate of users per title, which appears surprisingly similar to the platform revenue ratios seen in Humble Bundle promotions.


It's hard to say whether that correlation has any meaning, but at the very least, this users-per-title view sheds an entirely different light on Linux representation within the Steam Hardware & Software Survey results.


Whilst this article is probably a bit more theory heavy than my earlier Humble Bundle articles, I hope it's still of some interest and I'd love to hear any feedback or thoughts that people have :)

Enjoy! Article taken from GamingOnLinux.com.