Categories & Tags
Archive

On Web Player Regression Testing

January 12, 2010 in Technology by

One of the unwritten commitments we make to our customers is that our webplayer will play back their content identically in all versions we publish.  We constantly tweak and improve the webplayer. For example webplayer 2.6 added threaded background loading and improved animation culling. Each change that we make intending to improve the webplayer has the potential to break a customer game, and we don’t like that.  Remember, webplayer 2.6.1 can play content authored with Unity 2.0, 2.1, 2.5 and so on.  To make sure that we don’t break customer games with new webplayers we have to do a shed load of regression testing.  Lots and lots and lots.  So much so we have built a dedicated test rig to do the testing for us.

The regression rig has a server and multiple clients. The clients are PCs and Macs with our internal daily build webplayer installed. A server has a big list of our customer games, and sends jobs to each client that basically say “play this customer game”. Through a little bit of magic the clients take screen shots every second that the game runs, and these screen shots are forwarded onto the server as a record of the test execution. 

Currently we have nine clients (7 windows and 2 mac) in the rig.  Seven of these are physically located in our UK “test room”, the other two machines are under developers’ desks.  We are slowly adding more machines. When a webplayer is installed we get a snapshop of the machine configuration.  Using this database we can make sure we have the most popular machine specs represented in the rig.

Photo of the regression rig

Photo of the regression rig

Obviously, the games require inputs such as keyboard and mouse entry, random numbers, assets fetched from remote websites.  These inputs are all faked up when the regression rig runs.  What we do when we first get our hands on a customer game is give it to one of our testers who plays the game for us on the rig using the latest released webplayer.  In this manual first-run mode we capture all the inputs and store them in an input file.  Once we have this, we then generate a “golden” set of screen shots and know that these inputs generate these images.  The idea is that we now know what images the game should generate with a given set of inputs.  The regression rig can then compare images obtained from a customer game played back using the daily build webplayer against the golden set and we can tell quickly when one of our talented developers has broken something.  When we find we have broken the webplayer we can iterate back through code checkins looking for the commit that first introduced the problem.

The regression rig should help us find problems with the webplayer before you do, but as always, please let us know (by sending in a detailed bug report) if you have issues when you publish your game to the web. Oh, and for the record we do have people employed to do testing, it is not all done with the rig.

Share this post

Comments (4)

Comments are closed.

Jeiel
13 May 2010, 1:24 pm

How are you driving the mouse/keyboard input replays? Is there a program external to unity that plays back the keyboard mouse events? I’m trying to make a similar system inside Unity for recording and playing back tests but I’m having problems with the unequal framerates between my recording and the playback.

If you’re using an external program for keyboard and mouse events, can you please point me towards the program used by Unity to do this? Thanks!

13 May 2010, 2:18 pm

We have special build of Unity Web Player that can record all input events into a “journal” file. It also records all time durations; results of WWW requests etc. Then when replaying, we read the input, time, and other stuff from the journal. Of course, this way replaying does not catch regressions in input handling, for example.

Neel Sapre
2 Nov 2010, 11:36 pm

I’m very impressed with your QA setup. I’m trying to automate as much testing as i can. One question what is the % of automated and manual testing as far as new releases of unity web player is concerned. Also how long does it take to test the daily build of your web player?
Cheers!

Joe Riley
10 Nov 2010, 3:33 pm

Any plans on releasing that special build for testing purposes?

Leave a Reply

Comments are closed.