Committed to Quality

July 17, 2013 in Technology

UNT_blog_qa_post_20130709

Recently our CEO David Helgason gave a presentation at Unite Nordic where one of the key points was that Unity is having a huge focus on quality. As always, backing it up by numbers is my personal pet peeve, so here is little breakdown of how David is putting his money where his mouth is.

When I started in December 2011 the QA department in Unity consisted of 7 people in Denmark, England and Lithuania. What the exact cost of that was in 2011 I don’t know, but it is fair to say it is quite a bit lower than today.

In the budget for Unity in 2013 the QA department has 35 people with all of the salary, office space, furniture, hardware, lab hardware, training and travelling that comes with that. A handful are student workers on part time, but the grand total comes to 63360 hours of QA work a year. We have 70 developers, so it is an exact 1:2 tester-to-developer ratio which was the goal I stated to the company when I joined 18 months ago. You can see how I got to that number in a post I made on my personal blog. At 70 developers it also means that we are spending an additional 905 hours per developer per year in the quest to achieve great quality.

So what do we use all that QA for?

First of all we have manual testing. We use a lot of time on testing each feature before it ships, so the iteration and feedback cycle internally has been done before we hand it over to our community groups. And the feedback from those is also reproduced and verified. Claus has already made a blog post about the exploratory testing we do, which is yet another attack we do internally. But every area of Unity has a manual test engineer assigned to ensure we get coverage on all of it and catch as much as possible before we ship a version.

There is also a ton of automation going on. Kasper recently posted about our runtimetest framework, which has brought incredible results as we are increasing the number of supported platforms in Unity. Not only that, we are also shipping these tests to Google which they run in their test rigs on new Android versions, so they don’t accidentally break Unity in future versions of Android. And we are currently exploring similar collaboration with other platform holders.

Finally we’re going through the incoming lines of bugs our users are submitting. As the user base has grown, we are currently unable to answer all of them, but we are committed to get to a point where anyone who doesn’t just press the submit button actually gets a human reply.

All of this work is done to help you, the user, to get a better experience when you use Unity. The speed with which we are adding features is jaw dropping and unless we are vigilant about keeping our quality bar high, we can end in a bad state quickly. Fortunately, the company really gets it and there’s support for this effort from all sides.

 

Comments (12)

Subscribe to comments
  1. Gregory Booth

    August 2, 2013 at 11:59 pm / 

    Thanks UT.

    Having worked with Flash for the past 2 years and previously both Irrlicht and Ogre engines I’d have to say even with a few bugs Unity3D rocks like nothing else out there that I’ve tried.

    Rather puzzled when reading vociferous complaints since there isn’t anything I’ve seen that beats Unity3D with respect to their tech, support or their business model.

    Is there something better? ,,,then go use it.

  2. Thomas Petersen

    July 18, 2013 at 9:24 pm / 

    @The guy you treat…: All features are a crawl-walk-run effort. Everything has to evolve from something and for many features we prefer to start small, add good value and then improve on it. This includes NavMesh which has been improved a lot already.

    @JP: We already have quite a few people working from Canada. Check our people page to see all the nationalities we have: http://unity3d.com/company/people. As for the whole can and cannot say in public, we just believe we are doing a job good enough for us to be proud of it and letting the world know.

    @Justin: Yes, crowd sourcing is an eternal wish for us as well. We want to do more of it, have “things” in the pipeline for it, but there are quite a few obstacles to overcome as well. Stay tuned, we might surprise you down the road.

  3. Justin Link

    July 18, 2013 at 9:09 pm / 

    In my opinion, the biggest thing UT can do to improve its features (and documentation of them) is crowd source their development and QA. There are tons of features in Unity that are either lacking or undocumented that plenty of people within the community not only have the capability to address, but the willingness to as well. I think the Asset Store is great for game content and niche editor functionality, but it is being used a crutch for many unfinished features (GUI, for example). Create a system where the community can help the thing they use and depend on and Unity will be in a much better place.

  4. JP

    July 18, 2013 at 3:47 pm / 

    As a developer of some other software many people consume, I feel your pain and joy. There is so much you want to fix, show off, and implement and there are only so many hours before every release and financial cycle.

    My employer went publicly traded recently and it has meant a lot of changes on what we can and cannot say publicly. We get a lot of “why isn’t this fixed” and “when will this be fixed” and “this feature makes no sense” (it made sense for reasons we can’t make public!)

    Good job on expanding the QA team. Fogbugz isnt bad, it just needs a better “assetstore” ;-)

    Now if you’d just open a Vancouver BC office…

  5. Ashkan

    July 18, 2013 at 9:50 am / 

    Unity’s number of crashes and instabilities really decreased from previous times so you are in the right direction.

    Keep on doing the good job.

  6. Georges Paz

    July 18, 2013 at 4:37 am / 

    Unity is growing at light speed! Keep improving and pushing Unity forward, you are doing a great job!

  7. The guy who you treat like crap continuously

    July 17, 2013 at 11:40 pm / 

    “The speed with which we are adding features is jaw dropping”

    Yes, we’ve noticed that Unity. You generally implement a feature, throw it out there poorly implemented, then you fail to support it for several versions. One example of this is your poorly implemented NavMesh. Which in an email back to me you admitted it was lacking/poorly implemented and told me to “deal with it” and “what you see is what you get” and I should “just buy something from the asset store.” Good to know you guys know that.

    Meanwhile you keep adding more and more features. Hell a few weeks ago, one of your developers on stage admitted that you guys added a feature no one has asked for, and it was something that was more prevalent in the 1990s, but you just “felt like adding it.”

    IMO, you’re priorities are out of whack. You’d rather throw the asset store in our faces, give us poor subscriptions, and then tell us to “deal with it” when we have an issue with a feature you half assed.

    This post IMO is just justification for your lack of dedication to your features, and dedication to shoving assets down our throats. 30% profit is easier than fixing/implementing good features.

  8. Greg Quinn

    July 17, 2013 at 11:10 pm / 

    Unity’s upgraded QA has definitely reaped some benefits! On some of the issues we have been having with the Windows Store Beta, we have had day to day feedback from your QA team and have managed to isolate some very strange bugs indeed in a few days or less. Keep the quality bar high and UT will continue to soar sky high!
    http://monstertruckracingarenas.com

  9. Martin Schultz

    July 17, 2013 at 9:16 pm / 

    You’re doing a great job! (and your team of course!)

  10. Thomas Petersen

    July 17, 2013 at 8:05 pm / 

    @Noise: We’re using Fogbugz internally and as a bugtracker it is ok. Not spectacular, but not bad. The question of making the bugs publicly available is something we want to achieve for many, many reasons. When it will happen, though, I can’t say.

  11. Noisecrime

    July 17, 2013 at 7:15 pm / 

    I just wish Unity would move to a better bug tracking program and deal with the issues around making bugs public, or at least more public than they are now. Its so frustrating to waste hours tracking down problems to discover they are a bug and one that has probably been submitted already.

    Really its the bug tracker that I feel is letting down Unity. Its not searchable and has very poor selection of categories in which to attribute bugs to features. Having the title of the bug scraped from the first line of your bug report doesn’t help either. Hopefully it is something that will be improved over time as I can’t see QA being that happy with it either.

  12. Ashish Verma

    July 17, 2013 at 11:59 am / 

    DAVID (my virtual online inspiration asset) and Unity Team from devs to testers to sales & all.. (my virtual team friends). I want to wish you Many Many Thanks for keeping this kind of eagle eye on the day to day (sorry hour to hour work) and making a Tool that everyone loves. :) :)

    Don’t know why some of the people on others forums are complaining about the delay in the release of Unity 4.2. Yeah everybody including me would like to see that as soon as possible, and would like to see it at the same time as a bug free and a big update. And thats true that the perfection and commitments you mention prior to releasing a product are to be fulfilled in real. There is no sense of releasing quick updates with some pending buggs or laggy functionality and all, rather we should wait to see the new updates that are all perfect shots, bug free, useful and interesting ones.
    And thats what the Unity team is putting their hard-work , breaking their back, burning their eyes,
    and all dedicating to this aswome Tool “UNITY” we us (the fellow developers) ;)

    Everybody should appreciate it. All the Best to David, Unity Team and UNITY itself. :)
    Good luck n keep growing stay famous stay on top. :)

    -

    Ashish (Independent Unity Developer)
    I just started walking, and there are miles to go..
    Santa In the Valley of Gifts on ios, android, mac, and now on Kongregate.
    https://itunes.apple.com/us/app/santa-in-valley-gifts-s.i.v.g/id581665878?mt=8
    My second game Real Maze – 3D Augmented Reality Maze game coming soon in 1-2 weeks.
    Love you Unity :)

Comments are closed.