Ars Technica Benchmarks Show Windows 10 Beating SteamOS Performance

My first benchmark is that thanks to Linux and Steam on Linux, I can run games on PCs which can barely run Windows (**** planned obsolescence). My other benchmark is all my friends have Windows PCs and Mac and are very prouds but when it comes to play video games they drive miles to my place :)

I don't see problem here - this is at least some objective journalism. Yes, lot of ports have compromises cooked in and SteamOS isn't in it's best shape yet. As I believe in slow burner improvements regarding this, I just see it as another task to tackle for developers.

I personally don't think Vulkan is any kind of salvation here. Binary drivers just has to get better. Overall infrastructure has to get better. Also specific Steam Machine performance matters. People won't install SteamOS on their old boxes.

We need GOL Benchmarks (Windows VS SteamOS). I don't trust Ars Technica.

I think putting majority of blame on porters is kinda unjustified. OpenGL is simply lacking in performance and capability department when it comes to game development

i really don't see how people still use this as an arguement, it's all down to the drivers
some of the best performing games i've seen in my life have been openGL games, just look at Rage from id-software. which runs at 1080p60fps on Intel HD graphics, yes intel HD4000 graphics, also my mothers laptop is 7 years old and has a budget AMD card yet it still runs a beautifull game like Amnesia: The Dark Descent fine on it's native 1600x900 resolution. no directX game with those graphics come even close to that performance. openGL has always allowed for better optimization by the developers than directX (where games often require driver patches to improve performance), while it is harder to develop games with, it is not worse than directX performance wise

We need GOL Benchmarks (Windows VS SteamOS). I don't trust Ars Technica.

I don't plan to install Windows at any point, sorry.

I personally don't think Vulkan is any kind of salvation here.

It's not a guaranteed salvation but there's a good chance it will help to get some more AAA games to SteamOS.

Binary drivers just has to get better.

Yeah, but here lies exactly the big opportunity.
Vulkan drivers are significantly lower level and slimmer than OpenGL drivers.
Hence, easier to write and test.

However, much responsibility and control is transferred to the application developers.

But that is exactly what the AAA-Million-Dollar-Blockbuster-Developers were asking for.
Maybe not so much your creative indie buddy - but they can still use OpenGL.

TL;DR

Vulkan has the potential to make a difference.

I personally don't think Vulkan is any kind of salvation here. Binary drivers just has to get better. Overall infrastructure has to get better. Also specific Steam Machine performance matters. People won't install SteamOS on their old boxes.

Actually,I think that Vulkan does matter : its main objective, like the one of the other next gen APIs, is to take the optimization responsibility away from the driver, and put it back in developer's hands. On the plus side, extremely good drivers are no longer needed. On the other side... Bad developers will roll out even worse games (and they should stick with OpenGL, IMO). We could even see some game-specific Vulkan shader optimization in the proprietary drivers, if game developers are unable to code/test/debug properly.

Also, someone could turn a steambox into a lovely retro console :)

Edit : sub, you beat me by 2 minutes ;)


Last edited by MayeulC on 14 Nov 2015 at 2:11 pm UTC

However, much responsibility and control is transferred to the application developers.

But that is exactly what the AAA-Million-Dollar-Blockbuster-Developers were asking for.
Maybe not so much your creative indie buddy - but they can still use OpenGL.

It's going to be more like engine devs having that responsibility and control over performance and stability. Some engines are built for some games specifically, while others (unity, unreal, what have you will have to work on vulkan though) will be a great help for 'your creative indie buddy'. Which could mean specific engine optimisations will become more important, and drivers could become less important.

Ehm, I don't know how you can go more lower level than the most basic OpenGL calls. Don't forget that 90% of the OpenGL API is in fact redundant and if you look at the cleaner APIs such as OpenGL ES you will see that most of this stuff is in fact gone. I can imagine Vulkan making it easier, but not faster. I would again start with just the most basic problem, which is that they are using a distribution which is inherently slower than the top of the line such as gentoo and arch. And then drivers architecture, keep in mind that nvidia has used it's own architecture since its inception, so I'd guess it's probably optimized.

Now I'm thinking about it, I do have a kernel 4.x.x issue that actually causes games to drop frames every number of seconds. If he was testing with that kernel and having the same problem then the framerates are going to look skewed as well.

The fact is that benchmark is badly done, with a very old rig that no hardcore gamer who care about performance would have, with nonsense resolutions since SteamOS is dedicated to run on TVs and without giving much informations about the actual SteamOS version used or the methodology. I don't contest the fact that some game, especially AAA games have not as good performances as they have on Windows, I only contest those exaggerated results coming from an (intentionally) unprofessionally done benchmark, the wording and the conclusion obviously.

Steam Machines aren't high-end machines aimed at hardcore gamers, using weaker hardware is appropriate. I also usually like to point out to Windows users how much leaner and less resource-hogging Linux is. So on weaker hardware Linux should have an advantage because less of the limited resources is taken up by the OS.

I don't know what made them choose that particular resolution they used for Shadow of Mordor, but I highly doubt that would suddenly completely change the results. Maybe they just tried to use the highest resolution that was still somewhat playable at highest settings. And from what I remember, the 50% lower framerate is in line with results I've seen before. So any tweaking they might have done would make a few % difference at most, which really doesn't matter all that much once you're at -50%.

They don't mention the exact kernel and driver, I agree they should have, but they mention that they updated the system. So I guess what they have installed is whatever is in the current (or last week, after the launch) SteamOS repositories. If updating SteamOS didn't result in an up-to-date system, then yes, that would be unfair if they used an ancient driver, but it would also point to a different problem: SteamOS is supposed to be as easy to use as a console, you're not supposed to have to know that you even have a driver (of course, someone working for Ars should know). If using the automatic updating doesn't update the system properly, this needs to be fixed. So either they're on current drivers, or there's a problem with SteamOS.

Where I do agree is that using Shadow of Mordor as an example makes it look like Linux performs worse than it does on average. But they state that the selection criterion was a graphically intensive 3D game from 2014, which limits the selection Linux. SoM having an inbuilt benchmarking tool probably helped the choice as well. And we (the Linux community) can't on the one hand use SoM as an example of how we have recent 'AAA' games because it makes Linux look good, and on the other demand it isn't used for benchmarks because it makes Linux look bad.
The article did go on to test Valve's own games, so you could argue that they were actually trying to find games where Linux has a comparable performance. And they mention that games built from the ground up for Linux+OpenGL perform better on Linux.

I agree that the article's conclusion that Valve should improve the performance is completely wrong, we all know it's not up to Valve to fix nvidia's drivers. And that makes it all the more frustrating, since there's nothing anyone can do other than wait and hope for nvidia/amd to get their act together. But for nvidia/amd this likely has a low priority, since the user base is small. And the user base will remain small, because people will point to the poor performance on Linux, so Linux will remain low-priority... Catch-22.

This got longer than expected, I just wanted to write a quick comment...

Arse Technica. There, I said it, for no special reason.

I'll also add that I'm really disappointed in Valve's marketing, or complete lack thereof, of SteamOS. Over in /r/linux_gaming the meme going around is that "Valve is playing the long game, just like the launch of Steam itself, there's nothing to worry about guys!" The launch of Steam and SteamOS are two totally different situations, so I do wonder how well this strategy is going to work out. It really seems like Valve doesn't care if SteamOS totally flops because they're still rolling in dough either way.

It's in line with what Valve themselves stated as their plans for SteamOS and Steam hardware.
From an [IGN interview](http://www.ign.com/articles/2013/11/04/how-valve-engineered-the-perfect-controller?page=2):

"Usually, when a platform like this gets brought out, it’s a very different working method and proposition to customers,” he continued. “It looks more like a team that’s much larger than us has worked at perfecting something and finishing it, and then reducing risk as much possible and locking down that design, making it ready for a massive initial manufacturing push, spending billions of dollars on marketing. Very different from what we’re trying to do. We don’t have to be so risk-averse. We intentionally are operating this way because we think it will result in a much better product, in the short term and the long term, to be public about this, and to have it iterated with us and with partners and with users. But it lets us start small and grow over time.”

From a [Guardian interview](http://www.theguardian.com/technology/2015/oct/20/steam-machine-living-room-pc-gaming-valve):

“We built Steam for the same reasons we did all this: it didn’t exist, a bunch of people wanted it to exist, no one was building it, so we built it. There was this post-rationalisation of Steam after it was launched; like ‘of course you’d build that, it’s obvious’ – but, at the time, when we were building it, no one else was,” says Valve’s business development specialist Erik Johnson.

“Who knows if it’ll be the same this time, but if you ask customers, ‘do you want to be able to play all your Steam games in your living room with a controller?’, they’re like: ‘yeah, that sounds cool.’ That’s good enough for us to move forward. We’re not obsessed with what market we’re creating or what users we’re cannibalising from other platforms. No one ever makes good decisions based off that kind of data.”

And as you say, they can afford it, they have the money for this kind of approach.

Right now most ports run worse on Linux, a lot of it is down to OpenGL...

This is false. Games that are designed to be cross platform from the beginning run just as well on Linux as they do on Windows. For example, the performance between the Linux, OSX, and Windows version of X-Plane 10 is identical across all platforms. Victor Vran is another great example that has no significant performance difference between platforms. Not to mention all the Source Engine games.

The real problem is that porting from DirectX to OpenGL introduces performance loss because of how each system handles certain things, and there's not a lot porters can do about it without rewriting a game's graphics engine from scratch, which really isn't a viable solution.

The sooner we can get more developers thinking cross platform from day one, the sooner the performance gap between Linux and Windows will shrink.

The other thing to keep in mind is that even with the reduced performance, performance in Linux is still more than acceptable and won't negatively affect anybody's gaming experience. That's the problem with looking at benchmark numbers, because you'll see Windows getting 200FPS and Linux getting 100FPS, and everybody says, "Oh my gosh! Linux has 50% of the performance of Windows!" but does it really matter when everybody locks their refresh rate to 60Hz?


Last edited by Mountain Man on 14 Nov 2015 at 3:35 pm UTC

Seriously WTF? Why do they benchmark with a GTX 660 and a dual core processor at a resolutions of 2560x1600 and 1792x1120? Doesn't that look like they tested all combinations and choose the one that was worst for SteamOS. They have a GTX 980 Ti benchmark PC (<a href="http://arstechnica.com/gaming/2015/09/building-the-ultimate-x99-gaming-and-benchmarking-pc/" target="_blank" rel="noreferrer" style="cursor:help;display:inline !important;">http://arstechnica.com/gaming/2015/09/building-the-ultimate-x99-gaming-and-benchmarking-pc/</a>) why didn't they use this one?

Also, the processor Intel Pentium G3220 have integrated graphics and SteamOS does not support multiple GPUs, if it's active that would explain the bad performances.

Also there isn't a 4.1.0-0 version of SteamOS, and the NVidia drivers 358.91 are not available right now on SteamOS, so this test could have been made ages ago because we have no information whatsoever on when it was made and on which version of SteamOS.

Also I would add that the guy who wrote this is not clean at all, he have been part of a scandal involving gaming journalists of big gaming sites colluding together on how to shape the opinion by choosing what to cover and how to cover it in a private mailing list : <a href="http://www.breitbart.com/london/2014/09/17/exposed-the-secret-mailing-list-of-the-gaming-journalism-elite/" target="_blank" rel="noreferrer" style="cursor:help;display:inline !important;">http://www.breitbart.com/london/2014/09/17/exposed-the-secret-mailing-list-of-the-gaming-journalism-elite/</a> I should add that the guy that made the Polygon article is also part of this.

All valid points.

Even worse, it feels like they want to prove that Steam Machines with SteamOS are crap. And they're doing a damn fine job at this.

Yes, there does seem to be a pattern in the broader gaming press of protecting Windows from the unsavory horde of competition, from gleeful reports about Steam Survey results to disparaging comments about the lack of games compared to Windows to the recent articles about performance disparity. We can't deny the facts, of course, but so many of these articles seem to be written with a condescending smirk and a wink to the Microsoft faithful, and that really bothers me. It's like they're trying to kill SteamOS before it even has a chance.

The other thing to keep in mind is that even with the reduced performance, performance in Linux is still more than acceptable and won't negatively affect anybody's gaming experience. That's the problem with looking at benchmark numbers, because you'll see Windows getting 200FPS and Linux getting 100FPS, and everybody says, "Oh my gosh! Linux has 50% of the performance of Windows!" but does it really matter when everybody locks their refresh rate to 60Hz?

This may be valid for Portal, but if you want to run Shadow of Mordor on the Alienware SteamMachine, you'll have to make some serious compromises in the graphics settings to run it at 60Hz. I don't have the data, but the point remains that if you got the Alienware Alpha with the same hardware but running Windows, you'd get better graphics. And while that might not matter to you, it does matter to a lot of people, just look at any discussion comparing PS4 and XB1 performance.
Also, not everybody locks their games at 60Hz, I have a 144 Hz monitor (running at 120 Hz) and I like to put it to good use. Of course, I also have a better PC than an Alienware SteamMachine, so I'm good :D

Two things:

It's interesting that no one has mentioned the elephant in the room: Virtual Programming. Despite being a "wrapper", their eON system is now getting excellent performance on DirectX games. Perhaps that may be the best route for developers who aren't going to begin with native OpenGL code when they start making a game.

As for those developers who do start with OpenGL, the performance seems to be great. When developers don't rely on third-party porters, their games run very well on Linux. I'm talking about games like Victor Vran and Magicka 2.


Last edited by melkemind on 14 Nov 2015 at 4:31 pm UTC

Even worse, it feels like they want to prove that Steam Machines with SteamOS are crap. And they're doing a damn fine job at this.

Yes, there does seem to be a pattern in the broader gaming press of protecting Windows from the unsavory horde of competition, from gleeful reports about Steam Survey results to disparaging comments about the lack of games compared to Windows to the recent articles about performance disparity. We can't deny the facts, of course, but so many of these articles seem to be written with a condescending smirk and a wink to the Microsoft faithful, and that really bothers me. It's like they're trying to kill SteamOS before it even has a chance.

Except that I don't really think it's about protecting Windows, or holding a particular grudge against SteamOS, but rather a closedmindedness, and maybe a preconception about Linux being less professional or something?
Now, I need only blink in the direction of scientific communities to debunk that preconception, but most people have a tendency to protect their preconceptions and grope at things that affirm them.

Ehm, I don't know how you can go more lower level than the most basic OpenGL calls. Don't forget that 90% of the OpenGL API is in fact redundant and if you look at the cleaner APIs such as OpenGL ES you will see that most of this stuff is in fact gone. I can imagine Vulkan making it easier, but not faster. I would again start with just the most basic problem, which is that they are using a distribution which is inherently slower than the top of the line such as gentoo and arch. And then drivers architecture, keep in mind that nvidia has used it's own architecture since its inception, so I'd guess it's probably optimized.

Keep in mind what I am going to say applies to Mantle (since I have more information about it), but also to Vulkan since it carries its legacy.
Actually, drawing a triangle with one of these APIs requires no less than 800 lines of code. For a triangle !
The developer is now responsible for low-level resource management/allocation, like memory. With previous APIs, you just kept uploading textures to the memory, until it was full, with little idea of what was remaining or how the driver managed the calls. Now, you can have a much finer control on resources, allocating what's needed, and minimizing copies, or keeping objects in memory for later.
The game developer is also responsible for draw calls, and any other form of communication with the device. Which means he can make those in batch, saving precious CPU time, and (maybe more importantly) in a multithreaded manner (provided he can manage the added complexity of multiple threads, that is.) Oxide games tested their game ashes of the singularity on a simulated infinitely fast GPU (I can't find the link to their article, unfortunately). They found that DX12 (which is close enough to Mantle/Vulkan) was capable of using all the CPU cores at 100%, whereas with the older API, only a core was always at full load. It has to do with the way OpenGL (and I think DX<12 too) uses their command buffer to send commands to the GPU, with the driver often running only on one core.
So, the area where the new APIs really shine are in CPU-bound scenarios, or when drawing multiple objects on screen (a lot of drawcalls). There are a few more points which are in favor of Vulkan, like built-in compute and multi-GPU capabilities, etc.
For more information, take a look at the DX12 demos. They should be quite representative of Vulkan, IMO.
There's also a handful Vulkan demos available (well, at least the results).
View video on youtube.com
View video on youtube.com
One added benefit of Vulkan is the simplicity of drivers. While that's not completely relevant with Gallium3D and such, you don't have to implement the whole OpenGL spec; since you let the developer take care of it.
Of course, the downside of all this is that it comes with added complexity for the developer. Moreover, Vulkan might not offer any substantial gain over OpenGL in some applications. On the plus side, there will very probably be some helper libs to help devs abstract away the complexity of Vulkan (thus, more or less recreating a complete OpenGL in the application :P); Moreover, from what I gathered, it seems possible to make Vulkan calls within an OpenGL context, at least on nVidia hardware (the opposite is probably possible too, but I didn't check).
So yeah, it looks somewhat like the golden apple of Linux gaming, and it's very promising for embedded systems and engine developers too.
I'm sure I forgot something, but there's really a lot to say about it :)

Even worse, it feels like they want to prove that Steam Machines with SteamOS are crap. And they're doing a damn fine job at this.

Yes, there does seem to be a pattern in the broader gaming press of protecting Windows from the unsavory horde of competition, from gleeful reports about Steam Survey results to disparaging comments about the lack of games compared to Windows to the recent articles about performance disparity. We can't deny the facts, of course, but so many of these articles seem to be written with a condescending smirk and a wink to the Microsoft faithful, and that really bothers me. It's like they're trying to kill SteamOS before it even has a chance.

Except that I don't really think it's about protecting Windows, or holding a particular grudge against SteamOS, but rather a closedmindedness, and maybe a preconception about Linux being less professional or something?

Now, I need only blink in the direction of scientific communities to debunk that preconception, but most people have a tendency to protect their preconceptions and grope at things that affirm them.

We're essentially saying the same thing.