Committed to Quality
Recently our CEO David Helgason gave a presentation at Unite Nordic where one of the key points was that Unity is having a huge focus on quality. As always, backing it up by numbers is my personal pet peeve, so here is little breakdown of how David is putting his money where his mouth is.
When I started in December 2011 the QA department in Unity consisted of 7 people in Denmark, England and Lithuania. What the exact cost of that was in 2011 I don’t know, but it is fair to say it is quite a bit lower than today.
In the budget for Unity in 2013 the QA department has 35 people with all of the salary, office space, furniture, hardware, lab hardware, training and travelling that comes with that. A handful are student workers on part time, but the grand total comes to 63360 hours of QA work a year. We have 70 developers, so it is an exact 1:2 tester-to-developer ratio which was the goal I stated to the company when I joined 18 months ago. You can see how I got to that number in a post I made on my personal blog. At 70 developers it also means that we are spending an additional 905 hours per developer per year in the quest to achieve great quality.
So what do we use all that QA for?
First of all we have manual testing. We use a lot of time on testing each feature before it ships, so the iteration and feedback cycle internally has been done before we hand it over to our community groups. And the feedback from those is also reproduced and verified. Claus has already made a blog post about the exploratory testing we do, which is yet another attack we do internally. But every area of Unity has a manual test engineer assigned to ensure we get coverage on all of it and catch as much as possible before we ship a version.
There is also a ton of automation going on. Kasper recently posted about our runtimetest framework, which has brought incredible results as we are increasing the number of supported platforms in Unity. Not only that, we are also shipping these tests to Google which they run in their test rigs on new Android versions, so they don’t accidentally break Unity in future versions of Android. And we are currently exploring similar collaboration with other platform holders.
Finally we’re going through the incoming lines of bugs our users are submitting. As the user base has grown, we are currently unable to answer all of them, but we are committed to get to a point where anyone who doesn’t just press the submit button actually gets a human reply.
All of this work is done to help you, the user, to get a better experience when you use Unity. The speed with which we are adding features is jaw dropping and unless we are vigilant about keeping our quality bar high, we can end in a bad state quickly. Fortunately, the company really gets it and there’s support for this effort from all sides.