I like numbers!
Time has come for a deeper dive into the number crunching that is also a part of being in quality assurance. I like numbers. I love spreadsheets and nothing beats Excel in the world of numbers. Numbers are themselves irrelevant, but the story behind any set of numbers is fascinating, fantastic and sometimes also a horror. I will dig a bit into the numbers we use in Unity QA, how we get them and what we use them for.
There’s one specific term I use throughout this blog post, which is an Incident. Whenever a user submits a bug report through the bug reporter in Unity, we call it an Incident. QA processes the Incidents by reproducing it and once we have a reproduction we convert it to a bug. This is important in the following.
Navigating in the dark
Developing software is a journey in unknown territory, especially when you are trying to invent new cool stuff which has not been done before, so you need to figure out how to tell whether you’re on the right path or not. Wandering aimlessly will get you somewhere, but if you have a target and a few tools, you can get to a specific place instead of accepting whatever you stumble upon.
Ask the right questions and numbers can help you find a way. Here’s some of the questions I have asked:
- How many incidents have been reported on a specific version?
- How many of those incidents have we processed?
- How often a specific version was opened (adoption)?
- How many bugs did we convert from incidents?
- How good were the bugs? Did they end in a fix or was it duplicates or by design?
- Which areas are we getting bugs in?
- Are we getting more incidents from a new version compared to an old?
These are just some of the questions I asked myself.
The hardest part of tracking numbers is to make sure the data quality of these numbers is good enough. Lots of issues arise once you try to dig into a source of data and our journey has been the same as usual. A very simple thing is to change the type of a Fogbugz case from a “Bug” to an “Incident” when you guys report it to us. This change enables us to setup a better process for QA to reproduce the incidents and turn them into bugs once we have a successful repro. This gives devs the comfort that a bug has actually been reproduced inside Unity and is not related to anything specific on the user computer and, equally important; we can now start to have confidence in our bug count.
This was just one example of how a small change in process can help you attain more certainty about your numbers. I will show some of this data further down in this post.
A more techie part of gathering data is to pull in data from different sources, process them, link them up and build a cube (a pre-processed database with a special format for very fast reporting) for analysis on it. Currently we grab data from Fogbugz and Google Analytics, where we can track all sorts of bugs statistics and correlate them with the analytics data on a date and version level.
The technical part of this is done on SQL Server with an analysis services cube and reporting services for standard reports and Excel for custom analysis. The details of this setup are beyond this post, but you can find plenty of information on this setup from other sites.
Show it to me
This chart shows the action we received on 4.0 beta 2 and 4.0 beta 3. The lines represent the number of times the editor was opened in each version and the bars represent the number of incidents we received from the beta testers, all on a day by day basis. We can use this to compare how the adoption is of a specific version is going and unfortunately these numbers we see in this chart are not what we had hoped for, which leads us to search for explanations to why our beta testers are not as active now as they were during 3.5; we must be doing something the wrong way for this to happen. The effect of having a smaller adoption and less feedback from our community is extremely real in the sense that we might have to prolong a release period if we don’t feel we have had enough testing done on the product.
With this report we can see the overall status of how often an editor has been started on a specific version. We focus a lot on how to increase the number of people using these beta versions, so when we saw a very, very limited amount of feedback and adoption from our alpha group, we pushed hard to get a beta out to the larger beta group to increase the test surface. Likewise we are very eager to make a public version because the first public beta of 3.5 has been opened more than 1 million times (it is still being opened 10.000 times every week!). The Good-Bad Bug Ratio tells us how many of the bugs we have converted which ended up being fixed vs. those that end up being By Design or Won’t Fix, a number we track to find really great guys in our beta testing program and sometimes hand out a reward to them.
One outcome of monitoring these numbers was that we realized something was wrong with beta three and after a bit of thinking it turned out we had forgotten to set the auto update script to go to the next beta version, so many were staying on 4.0 beta 1 instead of moving to beta 3.
This overview tells us how we’re doing on handling incidents coming from the bugreporter in Unity. Every time someone submits a bug report, we treat it as an incident which has to be reproduced by QA before we convert it to a bug and this overview can tell us if we are behind on the queues, how many were converted and how good the converted bugs were.
I monitor this regularly to make sure we get all of the beta feedback processed. It is the most important information we get from you guys out there, so we have to handle it.
Bugs per Area
This is the bug situation we have in 4.0 right now, stacked by priority. All these bugs are destined for resolution before we ship, although reality sometimes catches up. With this chart we see where we have big chunks which need to be mitigated. Sometimes teams will cry for help from other teams to get their area under control. Fairly straight forward stuff here.
We collect data about which operating systems are used for developing on Unity and sometimes the results are…interesting.
Windows accounts for 80% of all users using the editor.
We are effectively no longer testing the editor on Windows Vista because the adoption of this platform is so low, a decision directly derived from the above statistics. Not to be confused with the webplayer, which is still tested on Windows Vista!
During the last month users opened Unity3D on a Windows XP machine 414.221 times, amounting to about 14% of all Windows users, so we are still having this one in our test runs.
Wrapping it up
I’ve tried to paint a picture of some of the things we do here in Unity. Often we need to navigate in the dark, but wherever possible we try to shed light on the way. I hope you found it interesting and if you would like to join us, go to http://www.unity3d.com/company/jobs .