
A bit over a year ago, we released a blog post with performance benchmarks for Unity WebGL, to compare WebGL performance in different browsers. We figured it was time to revisit those benchmarks to see how the numbers have changed.
Microsoft has since released Windows 10 with their new Edge browser (which supports asm.js and is now enabling it by default) – so we were interested to see how that competes. Also, we have an experimental build of Unity using Shared Array Buffers to run multithreaded code, and we wanted to see what kind of performance gains to expect. So we tested this in a nightly build of Firefox with Shared Array Buffer support.
You are welcome try our updated benchmark suite in your own browser here.
Some notes on methodology changes from last year’s benchmarks:
- This is using an updated build of the benchmark suite built in Unity 5.3. You can download the project folder for this benchmark suite here, so you can experiment with it locally, or try it on other platforms.
- This version has been stripped of all of the artwork and eye candy we had in the previous version. They added no value to the benchmarks, and the programmer art used never looked good anyway. Stripping the assets makes the builds smaller, and, more importantly, allows us to redistribute the project folder (see the link above).
- We omitted the “Mandelbrot GPU” benchmark from the test suite, as that test would really only benchmark the GPU and not show significantly different results on different browsers – so it would skew the overall results to show smaller relative differences.
- We omitted the comparison to a native standalone build we did last year, as we found those results to be misleading, as we are often running different code on different platforms (like different shader implementations with different quality settings – and also different scripting engines).
- We tested EdgeHTML 12 (Edge 20.10240.16384.0), which was the latest version when we ran the benchmarks, and which did not have asm.js enabled by default – so we turned it on manually. By now EdgeHTML 13 is released, which has asm.js enabled by default.
Here are the overall scores of the different browsers running the benchmark suite on an Windows 10 machine with a 3.3.GHz i7 CPU and an Nvidia GTX 960 GPU. Firefox 45 with Shared Array Buffers has a greyed out bar, as this is an experimental build of Unity running on a nightly build of Firefox:
Here are the scores of the different browsers running the benchmark suite on Mac OS X on a Retina MacBook Pro 15” with an 2.6 GHz i7 CPU (so we can see how Safari compares against the other browsers):
Here are the detailed results for each benchmark on windows (results scaled so that Firefox 41 32-bit equals 1.0 for each test):
And on OS X (again, scaled so that Firefox equals 1.0 for each test):
Here are the overall results from the benchmark build we showed last year compared to today, to see how performance has changed since (running on Mac OS X on a Retina MacBook Pro 15” with an 2.6 GHz i7 CPU):
Finally, here is a benchmark of how long it takes for Unity content to start up. The bars below represent the time in seconds it took between opening the benchmark project and rendering the first frame in different browsers. This is running content from a local disk, so the time to download files from the network is ignored. Firefox performs caching of asm.js compilation results, which means that when the same content is run more than once, compilation can be skipped, and subsequent runs will load faster. So we are showing numbers for both cold (first run) and hot (subsequent run with loading asm.js from cache) runs for Firefox:
Some findings:
- Firefox 42 64-bit is currently the fastest shipping browser in most of the benchmarks. The 32-bit version of Firefox is noticeably slower than the 64-bit version.
- Edge, as a new contender in these benchmarks comes in second, with results close to Firefox (and actually faster the Firefox 32-bit) in most benchmarks. In benchmarks which stress WebGL rendering performance (Particles, Asteroid Field), Edge performs best of all tested browsers.
- Safari delivers performance comparable to Chrome, up from a year ago when it was significantly slower than Chrome.
- Internet Explorer 11 is far behind the pack in just about everything, and is too slow to be of much use running Unity WebGL content.
- Our Unity build with Shared Array Buffers significantly improves performance (by several times in some of the benchmarks). This gives an outlook of future performance gains to expect.
- Overall, in Firefox, we are seeing an 18% performance increase compared to our results from one year ago. Some of this is attributable to Firefox 41 running faster than Firefox 32, the bigger difference seems to come from improvements in Unity and the emscripten compiler. These results are from OS X, where Firefox has been 64-bit for a long time. On Windows, the improvement since last year will be better, due to the arrival of 64-bit Firefox builds.
- Looking at load times, most modern browsers are able to load the benchmark project in between 5 and 7 seconds. Firefox is able to cache asm.js compilation, which brings subsequent loads down to 1.5-2 seconds.
Entradas relacionadas
38 Comentarios
Subscribirse a los comentariosYa no se aceptan más comentarios.
Jan de Wittt
enero 28, 2016 a las 10:36 amI’m missing Pale Moon in the benchmarks….
Dave
enero 26, 2016 a las 7:58 pmIs there a reason safari was left out of the loading/reloading time benchmark?
Raitoning
enero 10, 2016 a las 12:31 pmI can’t run the benchmark in Internet Explorer 11 ( Inori version ), the WebGL player return an Out of memory error.
On Linux, without proprietary drivers and running Firefox, the benchmark «runs», I mean, the whole screen is messed up, but at least it runs. With proprietary drivers on, I got a wonderfull pinkish bubblegum screen, and nothing works.
Raitoning
enero 10, 2016 a las 12:53 pmSorry about double post, but the WebGL bench just started in Internet Explorer 11 (Inori version), it seems that the Out of Memory is pretty random…
My scores:
Mandelbrot Script: 15 169
Instantiate & Destroy : 10 806
CryptoHast Script: 46 588
Animation & Skinning: 34
Asteroid Field: 2 148
Particles: 13 010
Physics Meshes: 29
Physics Cubes: 38
Physics Spheres: 51
2D Physics Spheres: 103
2D Physics Boxes: 72
AI Agents: 250
Overall Score: 11 476
Aram
diciembre 23, 2015 a las 3:25 pmIn window 10 in Firefox 42 64-bit I get this error message during the skinned mesh animation:
https://dl.dropboxusercontent.com/u/55553379/Unity%20webgl/Firefox%2042%2064-bit.PNG
It runs fine on Firefox 42 32-bit and on Edge 12.
Does anybody else have this problem?
I have 24 gigs of ram, GTX770 with 4gb DDR5
Gregory Pierce
diciembre 22, 2015 a las 3:34 amGlad to see some benchmarks around this. I was especially surprised to see how the different browsers performed on my own hardware. I definitely get different results on Windows 10. I actually get Edge>Firefox>Chrome. Not entirely sure why though.
As far as performance, its getting to «good enough» for many things. I won’t be trying to push AAA content directly in the browser, but for things like social games and apps – its definitely pretty close to where I don’t care so much about the benchmarks anymore.
agates
diciembre 19, 2015 a las 1:38 amIs there an available demo built using the shared array buffers? I have a 24-core workstation I’d like to test scalability on.
KYK
diciembre 18, 2015 a las 11:24 amHow about mobile browser?
i tried a small project with just a ugui scroll view, only have 15 fps in release WebGL in iphone 6s safari.
Looking forward to see better performance in mobile browser :D
Shmerl
diciembre 17, 2015 a las 7:38 pmWhat about Firefox and Chrome on Linux? Without it, these results are pretty incomplete.
Jonas Echterhoff
diciembre 17, 2015 a las 8:13 pmThe biggest point of these benchmarks is to see differences in performance of the browser’s JavaScript engines. These should be fairly consistent between different OSes. The only reason we tested both OS X and Windows in this benchmark was that we wanted to show how Safari compares to the other browsers. Some of the differences in results may also come from differences in WebGL implementations – OS X and Linux should be similar there, as both are based on OpenGL, and underlying GPU drivers (but showing the differences between GPU drivers is not the point of this benchmark).
mochabean
diciembre 17, 2015 a las 8:14 pmHere’s Chromium 47 on Arch Linux with an i5-4460 and a GTX 970.
http://i.imgur.com/RBlpk7o.png
DTH
diciembre 17, 2015 a las 3:43 pmMy results on Firefox 43 64 bits, Windows 8.1 64bits, NVIDIA GTX 780 (358.87):
With Angle (D3D11?) : 67794
With «webgl.disable-angle» : 68138
Seeing your results, either something went wrong in 43 or I’ve done something wrong in my Fox config.
Renard
diciembre 17, 2015 a las 12:28 pmFirefox 44.0a2 / Linux
Results are meh: http://i.imgur.com/wyPFjSl.png
Shiba
diciembre 17, 2015 a las 6:08 pmkek
Jack
diciembre 17, 2015 a las 7:17 pmhttp://i.imgur.com/bqpiPK9.png
This is on Arch with NVIDIA’s (proprietary) drivers and default settings aside from the Arc theme. I get similar results on Openbox.
David
diciembre 17, 2015 a las 10:46 pmI heard that 2016 will be the year of Linux on the desktop. Honestly, any decade now.
Anonypenguin
diciembre 17, 2015 a las 11:37 pmFedora Linux 23
AMD A8-6600K APU
Firefox 42
54965 overall
The results published in the blog are basically useless, because of being normalized. Since they don’t give the numbers in a way that I can use to compare my own results to the published results, it is all meaningless. Even after you run it yourself, you get no comparative information, and the types of things tested are very different from each other. Presumably there is some sort of typical reason common to published benchmarks why this would be concealed. ;)
Jonas Echterhoff
diciembre 18, 2015 a las 10:45 amThe reason is not about concealing anything, but simply about making all the benchmarks fit in a single chart, as the numbers for each of them are vastly different. The purpose of this blog post is to show relative differences between different browsers on the same hardware, to show how different JavaScript VMs and WebGL implementations compete. For this purpose, absolute numbers are not needed.
Richard Ayotte
diciembre 17, 2015 a las 11:55 pmDebian Sid with Iceweasel running on a 4K monitor. Not bad.
http://i.imgur.com/zNJdNzV.png
Jaimi
diciembre 16, 2015 a las 8:50 pmOSX 10.11
Firefox 43.0
iMac 24GB Ram, 4GB 780m.
During the Particles benchmark, firefox throws up a dialog «Out of memory. If you are the developer of this content, try allocating more memory to your WebGL build in the WebGL player settings.»
Wilken Haase
diciembre 16, 2015 a las 3:16 pmI would also have liked to see results from linux in comparison. I guess we would still see some funny results heading in unknown direction, but knowing is always better than guessing :)
Lifeizstrange
diciembre 16, 2015 a las 3:06 pmSad that Firefox 43 and 46 was not tested.
Also It could be a very useful idea to bench also on Linux. The gpu drivers are really different.
And what about the layout engine of the future ? Servo + webrender +browser.html
Xeon
diciembre 16, 2015 a las 1:34 pmI took a look at webgl export in 5.3 .. safe to say i won’t be bothering with it, the output scene lighting was completely off from the web standalone, and performance of the scene even after changing settings to be as low as possible was just not worth it.
And the webgl compile time turned my pc into barely useable brick for way too long, no idea who’s bright idea it was to set all compiling processes thread priority onto anything but ‘below normal’ kinda stupid given there is no setting in unity to set the thread priority before it starts, probably the same programmer who set the lighting building processes to the same.
oh and to top it off the webgl version left a 70mb opengl.js with the total size for webgl export being 82mb. Unity provides no way of seeing what the hell its exported out. For comparison the web standalone export was 3mb.
Maybe in 2017 webgl might be worth checking out again right now its bleh, ofc UE current track of releasing decent updates with actual built in engine features as opposed go find and buy such improvements at the asset store might have me switching to that in 2016 instead.
lowenz
diciembre 16, 2015 a las 11:29 amFirefox 43 64 bit / Windows 10 64 / i3 530@4 GHz+8GB RAM / GeForce 750 Ti
My results:
Native WegGL+WebGL2 – 64590
Native WegGL – 61865
Angle D3D11 – 54857
Angle – 50575
FX
diciembre 16, 2015 a las 10:12 amhttps://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/
Please test Firefox 46.0 Nightly.
Caspy7
diciembre 16, 2015 a las 12:24 amBit funny this was released the day that Firefox 43 came out. :)
Jaroslav Stehlik
diciembre 15, 2015 a las 11:03 pmFirefox 43.0 Out of memory during particle system.
OS X EL Capitan
Mac Book Pro Retina
mid 2014, 2,5 Ghz, 16 GB RAM,
David
diciembre 17, 2015 a las 10:47 pmFailfox is such an awesome pile of burpware.
Arowx
diciembre 15, 2015 a las 8:33 pmGreat work.
Will Shared Array Buffers be mapped to C# threads so our own multi-threaded code can take advantage of them via IL2CPP?
Any news on progress with SIMD, WebGL 2.0, WebVR and WebAssemby.
Jonas Echterhoff
diciembre 15, 2015 a las 9:38 pm«Will Shared Array Buffers be mapped to C# threads so our own multi-threaded code can take advantage of them via IL2CPP?»
Initially, no. Arbitrary C# threads are harder to allow because we cannot walk the stacks in JavaScript to perform garbage collection, so we can only allow threads in a controlled context where we know when we can safely assume no GC objects are referenced on any thread’s stack at certain times.
Wasstraat65
diciembre 16, 2015 a las 11:59 amWhat exactly are those controlled contexts and certain times if I may ask? Does not referencing GC objects mean we can only use local value types in threads? For example, what would not work in a multithreaded WebGL environment that would work in a multithreaded standalone?
Jonas Echterhoff
diciembre 16, 2015 a las 12:05 pmBasically, with the current setup, it is not possible to have any reference to a managed object (ie a GC handle) on any stack when GC takes place (which is currently once at the end of any frame). So managed threads which run longer then the duration of a frame would not be possible (since this would probably rule out most use cases people are interested in, we would not enable managed threads at all, before we can solve this problem somehow).
Adam
diciembre 15, 2015 a las 8:11 pmI disagree with the decision to omit standalone results from the comparison.
It’s important to understand the performance hit you’re going to take by choosing WebGL over standalone. Which areas are significantly weaker? Which are nearly comparable?
Further, performance is increasing over time on standalone as well and it’s valuable to compare the changes on each platform. If performance increases on standalone are outpacing those of WebGL, that means the gap between the platforms is increasing even in the face of the WebGL improvements and will affect the decision to leverage the platform. Likewise, if the gap is closing, that makes a stronger case for WebGL.
I have to wonder if the comparison was omitted because it potentially paints WebGL in a more negative light, but imho the comparison is crucial to making an informed decision. Sure, we can do the benchmarks ourselves, but why omit the information when you’re already publishing results publicly?
Jonas Echterhoff
diciembre 15, 2015 a las 10:08 pmWell the point is that you can interpret different things into comparing numbers from different platforms as they may run things very differently, with different outputs.
But, fair enough – here are the numbers from my MacBook Pro:
Native:
Mandelbrot Script: 30754
Instantiate & Destroy: 69579
CryptoHash Script: 122416
Animation & Skinning: 1156
Asteroid Field: 10077
Particles: 127690
Physics Meshes: 1677
Physics Cubes: 2649
Physics Spheres: 3237
2D Physics Spheres: 3359
2D Physics Boxes: 2170
AI Agents: 6471
Overall: 143219
Firefox:
Mandelbrot Script: 133846
Instantiate & Destroy: 61892
CryptoHash Script: 212708
Animation & Skinning: 495
Asteroid Field: 8566
Particles: 102580
Physics Meshes: 600
Physics Cubes: 820
Physics Spheres: 1658
2D Physics Spheres: 2466
2D Physics Boxes: 1488
AI Agents: 1950
Overall: 84443
Noticeable points:
-Scripting benchmarks are actually faster in WebGL then in native. This is due to different scripting backends used (il2cpp vs mono)
-Benchmarks which are mostly rendering bound (Asteroid Field, Particles) perform very close to native.
-Benchmarks which benefit from multithreading a lot (physics, skinning) are significantly faster in native code.
Overall not much has changed in these findings since last year. No surprises there, as the constraints have not really moved (other then some browsers like Edge catching up on performance). This will change when technologies like Shared Array Buffers will be available.
Adam
diciembre 16, 2015 a las 5:14 pmThanks for the reply and the added information.
I agree there is room to interpret exactly where the performance differences come from. However, it’s not as though you can choose to build for «WebGL using the standalone code paths». Regardless of the sources of the differences, the fact is they are the inherent cost of choosing to target WebGL. As such, the absolute differences are the more salient information, rather than the precise reason they exist.
I’m happy to see WebGL coming along. Thanks for the blog post.
DaveVoyles
diciembre 15, 2015 a las 6:10 pmAwesome, I was in the middle of writing a presentation on Unity’s WebGL benchmarks when you updated this today.
Thanks for keeping us in the loop, looking forward to seeing how browsers improve perf over time.
Mal
diciembre 15, 2015 a las 5:54 pmGreat to see more info on WebGL builds.
What about mobile ( other WebGL tech seem to work ok, eg Blend4Web, PlayCanvas etc ), would be cool to have WebGL as a potential option for smaller games across desktop and mobile.
Also would be cool to have report on reduced file size ( if any ), of say a single cube, to get an idea of the overhead pre adding assets. Maybe build times as well.
Keep up the great work in this area!
Mal
Jonas Echterhoff
diciembre 15, 2015 a las 9:36 pmUnity WebGL runs on mobile, but currently the results are only usable on very high-end devices (depending on your content), so we cannot recommend it. The other engines you named are not really comparable to Unity in terms of functionality, and have a much smaller foot print in code size, which makes them a better fit for today’s mobiles. I expect technology to catch up with this in the future, both in the form of faster mobile devices with more memory, and better performance from browsers and technologies such as WebAssembly.
Unity 5.4 should give you an option for much faster development builds. We are also working on a Build Report feature which will get you detailed information on build size and where the size comes from (currently planned for 5.5).