Ars Technica Benchmarks Show Windows 10 Beating SteamOS Performance

My first benchmark is that thanks to Linux and Steam on Linux, I can run games on PCs which can barely run Windows (**** planned obsolescence). My other benchmark is all my friends have Windows PCs and Mac and are very prouds but when it comes to play video games they drive miles to my place :)

I don't see problem here - this is at least some objective journalism. Yes, lot of ports have compromises cooked in and SteamOS isn't in it's best shape yet. As I believe in slow burner improvements regarding this, I just see it as another task to tackle for developers.

I personally don't think Vulkan is any kind of salvation here. Binary drivers just has to get better. Overall infrastructure has to get better. Also specific Steam Machine performance matters. People won't install SteamOS on their old boxes.

We need GOL Benchmarks (Windows VS SteamOS). I don't trust Ars Technica.

Quoting: chopdokI think putting majority of blame on porters is kinda unjustified. OpenGL is simply lacking in performance and capability department when it comes to game development

i really don't see how people still use this as an arguement, it's all down to the drivers
some of the best performing games i've seen in my life have been openGL games, just look at Rage from id-software. which runs at 1080p60fps on Intel HD graphics, yes intel HD4000 graphics, also my mothers laptop is 7 years old and has a budget AMD card yet it still runs a beautifull game like Amnesia: The Dark Descent fine on it's native 1600x900 resolution. no directX game with those graphics come even close to that performance. openGL has always allowed for better optimization by the developers than directX (where games often require driver patches to improve performance), while it is harder to develop games with, it is not worse than directX performance wise

Quoting: WindousicoWe need GOL Benchmarks (Windows VS SteamOS). I don't trust Ars Technica.

I don't plan to install Windows at any point, sorry.

Quoting: PeciskI personally don't think Vulkan is any kind of salvation here.

It's not a guaranteed salvation but there's a good chance it will help to get some more AAA games to SteamOS.

Quoting: PeciskBinary drivers just has to get better.

Yeah, but here lies exactly the big opportunity.
Vulkan drivers are significantly lower level and slimmer than OpenGL drivers.
Hence, easier to write and test.

However, much responsibility and control is transferred to the application developers.

But that is exactly what the AAA-Million-Dollar-Blockbuster-Developers were asking for.
Maybe not so much your creative indie buddy - but they can still use OpenGL.

TL;DR

Vulkan has the potential to make a difference.

Quoting: PeciskI personally don't think Vulkan is any kind of salvation here. Binary drivers just has to get better. Overall infrastructure has to get better. Also specific Steam Machine performance matters. People won't install SteamOS on their old boxes.

Actually,I think that Vulkan does matter : its main objective, like the one of the other next gen APIs, is to take the optimization responsibility away from the driver, and put it back in developer's hands. On the plus side, extremely good drivers are no longer needed. On the other side... Bad developers will roll out even worse games (and they should stick with OpenGL, IMO). We could even see some game-specific Vulkan shader optimization in the proprietary drivers, if game developers are unable to code/test/debug properly.

Also, someone could turn a steambox into a lovely retro console :)

Edit : sub, you beat me by 2 minutes ;)


Last edited by MayeulC on 14 November 2015 at 2:11 pm UTC

Quoting: subHowever, much responsibility and control is transferred to the application developers.

But that is exactly what the AAA-Million-Dollar-Blockbuster-Developers were asking for.
Maybe not so much your creative indie buddy - but they can still use OpenGL.

It's going to be more like engine devs having that responsibility and control over performance and stability. Some engines are built for some games specifically, while others (unity, unreal, what have you will have to work on vulkan though) will be a great help for 'your creative indie buddy'. Which could mean specific engine optimisations will become more important, and drivers could become less important.

Ehm, I don't know how you can go more lower level than the most basic OpenGL calls. Don't forget that 90% of the OpenGL API is in fact redundant and if you look at the cleaner APIs such as OpenGL ES you will see that most of this stuff is in fact gone. I can imagine Vulkan making it easier, but not faster. I would again start with just the most basic problem, which is that they are using a distribution which is inherently slower than the top of the line such as gentoo and arch. And then drivers architecture, keep in mind that nvidia has used it's own architecture since its inception, so I'd guess it's probably optimized.

Now I'm thinking about it, I do have a kernel 4.x.x issue that actually causes games to drop frames every number of seconds. If he was testing with that kernel and having the same problem then the framerates are going to look skewed as well.

Quoting: NyamiouThe fact is that benchmark is badly done, with a very old rig that no hardcore gamer who care about performance would have, with nonsense resolutions since SteamOS is dedicated to run on TVs and without giving much informations about the actual SteamOS version used or the methodology. I don't contest the fact that some game, especially AAA games have not as good performances as they have on Windows, I only contest those exaggerated results coming from an (intentionally) unprofessionally done benchmark, the wording and the conclusion obviously.

Steam Machines aren't high-end machines aimed at hardcore gamers, using weaker hardware is appropriate. I also usually like to point out to Windows users how much leaner and less resource-hogging Linux is. So on weaker hardware Linux should have an advantage because less of the limited resources is taken up by the OS.

I don't know what made them choose that particular resolution they used for Shadow of Mordor, but I highly doubt that would suddenly completely change the results. Maybe they just tried to use the highest resolution that was still somewhat playable at highest settings. And from what I remember, the 50% lower framerate is in line with results I've seen before. So any tweaking they might have done would make a few % difference at most, which really doesn't matter all that much once you're at -50%.

They don't mention the exact kernel and driver, I agree they should have, but they mention that they updated the system. So I guess what they have installed is whatever is in the current (or last week, after the launch) SteamOS repositories. If updating SteamOS didn't result in an up-to-date system, then yes, that would be unfair if they used an ancient driver, but it would also point to a different problem: SteamOS is supposed to be as easy to use as a console, you're not supposed to have to know that you even have a driver (of course, someone working for Ars should know). If using the automatic updating doesn't update the system properly, this needs to be fixed. So either they're on current drivers, or there's a problem with SteamOS.

Where I do agree is that using Shadow of Mordor as an example makes it look like Linux performs worse than it does on average. But they state that the selection criterion was a graphically intensive 3D game from 2014, which limits the selection Linux. SoM having an inbuilt benchmarking tool probably helped the choice as well. And we (the Linux community) can't on the one hand use SoM as an example of how we have recent 'AAA' games because it makes Linux look good, and on the other demand it isn't used for benchmarks because it makes Linux look bad.
The article did go on to test Valve's own games, so you could argue that they were actually trying to find games where Linux has a comparable performance. And they mention that games built from the ground up for Linux+OpenGL perform better on Linux.

I agree that the article's conclusion that Valve should improve the performance is completely wrong, we all know it's not up to Valve to fix nvidia's drivers. And that makes it all the more frustrating, since there's nothing anyone can do other than wait and hope for nvidia/amd to get their act together. But for nvidia/amd this likely has a low priority, since the user base is small. And the user base will remain small, because people will point to the poor performance on Linux, so Linux will remain low-priority... Catch-22.

This got longer than expected, I just wanted to write a quick comment...