Moving the bug fight forward
Testing might not be a part of the romantic picture we all have about game development. Often it’s more seen as a dirty job that someone has to do. At Unity we have a growing team of test engineers who really love their job and take pride in achieving excellence, and I’d like to share with you a little bit about how we work. I am guessing that most of you have some kind of interest in us doing our job as good as possible and that you’d like to know what happens when you open the bug reporter and click ‘Submit’.
When you submit a bug report it goes into our Incoming Bug Queue. From there we do what we call sweeping. We weed out bug reports that contain nothing at all – no description, no information and no attached project, and we weed out the spam.
All other bugs go into our Investigation Queue. In there we look at each case and simplify it until we get the cleanest scenario possible where the bug happens. If the steps are too unclear for us to reproduce the bug, or if we need a test project, we reply back to you asking for what we need. It’s also often a matter of the user doing something wrong or needing some help. In that case we respond with the assistance needed and add an entry to the documentation or answers.unity3d.com if needed.
The cases from investigation that are actually real bugs are then moved to our Test Creation Queue. Now what does that mean – didn’t we already test it? Well, it means that we take the simplified reproduction case and implement it in one of our automated test frameworks, depending on where the case fits. We have a few of those including our awesome Regression Rig. When the test is written, it will continuously run on multiple platforms. Whenever new code and functionality is committed, it will be held up against the growing list of tests. In theory this ensures that once a bug has been fixed and added to the automated tests, it will never show it’s ugly face again.
Fixing and verifying
From the test creation queue we end up with either writing a test or in some cases where this is not feasible, we make sure that the steps to reproduce the bugs are crystal clear. Then we assign the case to development. One of our highly specialized developers will take the case, fix it and send it back to us for verification. At this point the test written should always pass and the test department also verifies the fix manually. If something is not right, it’s back to the developer until we are sure the bug is really fixed.
It has taken us a long time to hone these processes – Unity has grown immensely since the early days when there wasn’t even a Test department but just a handful of programmers making the product, testing it and providing support. Unity is now vastly bigger in terms of features, users and people working on the product. This also means, that there was a time when our capacity for handling bugs and support cases was lower and we didn’t have the same processes. The test team is bigger now and armed to the teeth with sharpened minds, rockstar programming skills and a work horse mentality.
One of the ways we’re moving along now is drawing a line somewhere in the backlog of cases from when our capacity was lower. We have a bunch of old cases where the majority are duplicates of already fixed cases, cases with no information or cases that are obsolete because of changes in Unity. We will be focusing on the newer bugs, but there will also be something in that old pile that still has relevance. Chances are that these issues will have been reported again with Unity 2.6 or 3.0, but if you are sitting with an old bug report you submitted, that you believe is still relevant then you can help us get to it by submitting it again.
We are aiming for a workflow where we process each and every relevant case that gets opened and provide feedback as soon as possible. I hope you’re up for the task of helping us by submitting high quality bug reports and that you will continue to have a little patience and love for the Test Team – cause we love you!