I like numbers!

August 1, 2012 in Community

Time has come for a deeper dive into the number crunching that is also a part of being in quality assurance. I like numbers. I love spreadsheets and nothing beats Excel in the world of numbers. Numbers are themselves irrelevant, but the story behind any set of numbers is fascinating, fantastic and sometimes also a horror. I will dig a bit into the numbers we use in Unity QA, how we get them and what we use them for.

There’s one specific term I use throughout this blog post, which is an Incident. Whenever a user submits a bug report through the bug reporter in Unity, we call it an Incident. QA processes the Incidents by reproducing it and once we have a reproduction we convert it to a bug. This is important in the following.

Navigating in the dark

Developing software is a journey in unknown territory, especially when you are trying to invent new cool stuff which has not been done before, so you need to figure out how to tell whether you’re on the right path or not. Wandering aimlessly will get you somewhere, but if you have a target and a few tools, you can get to a specific place instead of accepting whatever you stumble upon.

Ask the right questions and numbers can help you find a way. Here’s some of the questions I have asked:

  • How many incidents have been reported on a specific version?
  • How many of those incidents have we processed?
  • How often a specific version was opened (adoption)?
  • How many bugs did we convert from incidents?
  • How good were the bugs? Did they end in a fix or was it duplicates or by design?
  • Which areas are we getting bugs in?
  • Are we getting more incidents from a new version compared to an old?

These are just some of the questions I asked myself.

The Gathering

The hardest part of tracking numbers is to make sure the data quality of these numbers is good enough. Lots of issues arise once you try to dig into a source of data and our journey has been the same as usual. A very simple thing is to change the type of a Fogbugz case from a “Bug” to an “Incident” when you guys report it to us. This change enables us to setup a better process for QA to reproduce the incidents and turn them into bugs once we have a successful repro. This gives devs the comfort that a bug has actually been reproduced inside Unity and is not related to anything specific on the user computer and, equally important; we can now start to have confidence in our bug count.

This was just one example of how a small change in process can help you attain more certainty about your numbers. I will show some of this data further down in this post.

A more techie part of gathering data is to pull in data from different sources, process them, link them up and build a cube (a pre-processed database with a special format for very fast reporting) for analysis on it. Currently we grab data from Fogbugz and Google Analytics, where we can track all sorts of bugs statistics and correlate them with the analytics data on a date and version level.

The technical part of this is done on SQL Server with an analysis services cube and reporting services for standard reports and Excel for custom analysis. The details of this setup are beyond this post, but you can find plenty of information on this setup from other sites.

Show it to me

Daily Adoption

This chart shows the action we received on 4.0 beta  2 and 4.0 beta 3. The lines represent the number of times the editor was opened in each version and the bars represent the number of incidents we received from the beta testers, all on a day by day basis. We can use this to compare how the adoption is of a specific version is going and unfortunately these numbers we see in this chart are not what we had hoped for, which leads us to search for explanations to why our beta testers are not as active now as they were during 3.5; we must be doing something the wrong way for this to happen. The effect of having a smaller adoption and less feedback from our community is extremely real in the sense that we might have to prolong a release period if we don’t feel we have had enough testing done on the product.

Version Adoption

With this report we can see the overall status of how often an editor has been started on a specific version. We focus a lot on how to increase the number of people using these beta versions, so when we saw a very, very limited amount of feedback and adoption from our alpha group, we pushed hard to get a beta out to  the larger beta group to increase the test surface. Likewise we are very eager to make a public version because the first public beta of 3.5 has been opened more than 1 million times (it is still being opened 10.000 times every week!). The Good-Bad Bug Ratio tells us how many of the bugs we have converted which ended up being fixed vs. those that end up being By Design or Won’t Fix, a number we track to find really great guys in our beta testing program and sometimes hand out a reward to them.

One outcome of monitoring these numbers was that we realized something was wrong with beta three and after a bit of thinking it turned out we had forgotten to set the auto update script to go to the next beta version, so many were staying on 4.0 beta 1 instead of moving to beta 3.

Incidents Status

This overview tells us how we’re doing on handling incidents coming from the bugreporter in Unity. Every time someone submits a bug report, we treat it as an incident which has to be reproduced by QA before we convert it to a bug and this overview can tell us if we are behind on the queues, how many were converted and how good the converted bugs were.

I monitor this regularly to make sure we get all of the beta feedback processed. It is the most important information we get from you guys out there, so we have to handle it.

Bugs per Area

This is the bug situation we have in 4.0 right now, stacked by priority. All these bugs are destined for resolution before we ship, although reality sometimes catches up. With this chart we see where we have big chunks which need to be mitigated. Sometimes teams will cry for help from other teams to get their area under control. Fairly straight forward stuff here.

Operating Systems

We collect data about which operating systems are used for developing on Unity and sometimes the results are…interesting.

Windows accounts for 80% of all users using the editor.

We are effectively no longer testing the editor on Windows Vista because the adoption of this platform is so low, a decision directly derived from the above statistics. Not to be confused with the webplayer, which is still tested on Windows Vista!

During the last month users opened Unity3D on a Windows XP machine 414.221 times, amounting to about 14% of all Windows users, so we are still having this one in our test runs.

Wrapping it up

I’ve tried to paint a picture of some of the things we do here in Unity. Often we need to navigate in the dark, but wherever possible we try to shed light on the way. I hope you found it interesting and if you would like to join us, go to http://www.unity3d.com/company/jobs .

 

Comments (16)

Subscribe to comments
  1. Thomas Petersen

    August 6, 2012 at 12:23 am / 

    @Marvin: It was not my intention to say anything bad about our groups in this post. Whenever a beta or alpha version is having low adoption we can only blame ourselves for not doing this or that, and using some numbers can help us find the reason, or at least realize the problem soon enough to search for a reason.

  2. Marvin

    August 5, 2012 at 3:51 pm / 

    I believe the alpha numbers are skewed low because the main feature requested to be tested during the alpha was not “exciting”, was not asked for, and broke current workflows. However, the alpha test generated many excellent and well thought out posts about this main feature, which were reiterated in greater number during the beta, and resulted in good changes in the current beta. I am concerned Unity will base the effectiveness of alpha testing on this first run, which imho would be a mistake.

    Put the new GUI in the next alpha test, and i can guarantee more interest and thorough testing :)

  3. Chris

    August 3, 2012 at 3:05 pm / 

    While it was an interesting article, the XP and Vista chart at the bottom was probably the most interesting for me. It looks like a lot of people did like I did, migrating to Windows 7 straight from XP.

  4. AngryAnt

    August 3, 2012 at 1:10 am / 

    @Eli
    The beta group is currently very large and is occasionally pruned based on activity (IE silent beta testers not using the betas are off the list).

    @lewis
    No announcements yet. Stay tuned.

  5. lewis

    August 2, 2012 at 10:26 pm / 

    Thomas, when Unity 4 will be relased?

  6. Eli Curtz

    August 2, 2012 at 8:05 am / 

    I hope you’ve gotten a lot of new beta testers for 4.0. In my experience even the most dedicated beta testers get burned out (they are just volunteers, after all.) Many of the people who were the most valuable 3.5 testers might still be great community members, but very few of them will still be the best testers for the new version.

  7. Thomas Petersen

    August 2, 2012 at 12:49 am / 

    @Scott: It is not yet in a public beta. You will be notified when a drop is ready for you.

  8. Scott McKie

    August 1, 2012 at 5:41 pm / 

    The 4.0 beta is available? I’d be running it now, if I’d known! (I have already paid for the upgrade)

  9. Thomas Petersen

    August 1, 2012 at 1:58 pm / 

    @Kryptos: It’s because we have no active bugs on the linux player right now. So we’re waiting on bugs to be found…

  10. Kryptos

    August 1, 2012 at 1:46 pm / 

    Great article. But I’m surprised to find no data related to linux standalone.

  11. Thomas Petersen

    August 1, 2012 at 12:56 pm / 

    @Ashkan: This is something we would like to do, but haven’t even started prototyping it yet. But definitely a good idea!

  12. Ashkan

    August 1, 2012 at 12:39 pm / 

    Really great read! seriously awesome.
    Also showing bug ratios of normal usrs and maybe rewarding them and maybe opening the incidents collection to users to try to reproduce them and vote them up might help you to try more solid incidents first. of course it’s posible for public releases.

  13. Chris Park

    August 1, 2012 at 12:12 pm / 

    @Thomas: That’s kind of my point, though. If your numbers are partly related to the number of crashes that a given developer experiences — because that’s a key thing that causes application restarts — then your crash-prone releases will have inflated numbers and your rock-stable versions will seem to be unpopular by comparison.

    We’re still on 3.3 because there were some bugs in 3.4 that were causing our games to crash as soon as they launched, which was problematic. 3.5 is something we’re looking to move to soon (and we’ve already bought version 4.x), but some of the early problems with the multithreaded rendering pipeline scared us off a bit.

    Also the fact that PowerPC support was dropped makes it more tricky to upgrade our older games to 3.5, because we have customers who bought our games and play them on that platform. So we’re trying to get at least our older games onto a major new official release before we do the upgrade (we go through a cycle of betas and then officials). We’ve got quite a few customers, which makes it kind of a tricky thing to do that sort of thing quickly; and being in the middle of a intensive release cycles for several games made it so that we couldn’t really take time out to potentially have to hunt down some new engine quirks; sticking with the devil we know instead, so to speak.

    That makes it sound like I have a really negative view of Unity, which I don’t. But each new version comes with quite a few quirks and risks that have to be learned anew, and there’s only certain times of year when we can reasonably do that.

  14. Alex

    August 1, 2012 at 10:45 am / 

    Not the blog post i was expecting next but a very interesting read nonetheless.

    PS: Excel Forevah!

  15. Thomas Petersen

    August 1, 2012 at 10:15 am / 

    @Chris: The interesting thing about these numbers are not the individual number as such, but the comparison of them to previous versions and releases.

    How come you’re still on 3.3? I would urge you to upgrade to 3.5 as soon as possible.

  16. Chris Park

    August 1, 2012 at 10:02 am / 

    Very interesting read. Two things that might be affecting your adoption rate numbers that I don’t know if you’ve accounted for, however:

    1. In the case of myself and a lot of other developers I know, we tend to open our development tools once and then leave them running for days, weeks, or even months. Until we need to reboot or the dev tools crash, in other words. This would skew your numbers to being misleadingly low on adoption.

    2. On the other hand, the unity editor crashes a fair bit. Sometimes as many as 3-5 times per day on version 3.3, which is the version my company is still using at the moment. This would cause your adoption numbers on a crash-prone version to be misleadingly high; and thus a version of the editor that is substantially more stable might _seem_ to have vastly lower adoption despite not really having such.

    I guess as kind of a third note, I don’t even use the unity editor every day. Visual studio can build the DLL for the code of a game just as well as the unity editor can, except that visual studio is faster at it because it doesn’t reimport assets each time. Therefore, unless we’ve added assets (which we do comparatively rarely next to how often we update existing code files), then we’re often just compiling in visual studio and then running the standalone application with the new DLL attached. That, again, would skew your adoption numbers — but not by version of Unity at least, I wouldn’t think.

Comments are closed.