The Unity Engine QA Process
We often get requests to explain how we test the engine before we release it. We have previously blogged about it, but none of these posts give a clear overview of the entire process. It’s a large undertaking to do in just one post and for that reason this will be a series of blog posts.
This first post is going to outline the structure of our test efforts and subsequent blog posts will be made to describe details of each individual part, always linking back to this overview. It is also worth noting that every strategy comes from the product and deployment model, so our process on the engine is very different from the process we use on our services. This blog series relates only to the Unity engine itself.
In the simplest form, our releases go through the following lifecycle:
This is what is most obvious to our users, because our version numbering follows the same structure. With this frame we can start exploring what happens at every stage of the product from a QA point of view.
Before we even get code into trunk where we integrate all the features, we do testing on the feature in a branch based off of our mainline source repo, trunk. This is typically feedback on the usability of the feature and ensuring it doesn’t introduce massive instability. How this is done effectively is a topic for a blog post of its own.
Once the feature has been tested in-branch, the developers create the initial Pull Request (PR) in which a code review is done on the code, all automation has been executed and proven to pass, a review of the test cases written is done and the Release Manager does a risk assessment before merging it to trunk. A lot of this is automated through our merge queue process, but it can involve 20-30 people to get a big feature into trunk. And remember, this process is done on every single PR going to trunk, not just one time.. All of these are blog topics for later.
In the alpha phase, it is all about integration of all the features done in the branches. Before we send the first alpha to the alpha list, we run our Release Acceptance Test (RAT), which is a smoke test covering the surface of the platforms and the editor to make sure we have human eyes on as many parts as possible before we ship.
During the following alphas we get more branches trickling in and we continue our integration testing on the builds. During this time we also get a lot of user feedback on the usability and viability of the features from our alpha users. This sometimes results in a feature being cut completely from the release.
In the end of the alpha phase we have an Exploratory Test (ET) week. This approach was covered by Claus in 2013 and the format remains the same today.
The beta phase is now fully public, so we ship the builds directly to all users. Creating the engine with our community has been a key part of how we developed Unity through all the years and it is still an integral part of every release.
During the beta phase we do a Full Test Pass (FTP), which is a manual regression test pass of all our manual test cases, which test the various workflows and runtime features, spread across all the supported platforms, so we get as broad a surface as possible with human eyes and brains on all of it.
The FTPv2 is a special session we carry out to combine the workflows to make actual games with the build. It is a fun and intense way of testing Unity.
The Asset Store Bash is three days where we go through a list of Asset Store packages compiled from the top 500, upgrade the projects and see if they reveal problems in the builds.
The documentation bash is a few days where we go through our documentation in order to find bugs and discrepancies between the docs and the actual functionality in the engine.
We also execute upgrade tests, in which we take large user projects and upgrade them. It is surprisingly difficult to perform this test and we most certainly need a blog post explaining why this is so. We realize it is a source of a lot of user pain, so we invest a lot of effort in this.
The release candidate phase is started off by an RC Test Phase where we do a risk driven pass on the engine. This risk assessment is done by the dev teams involved in each area and we use this to drive a targeted effort to get more comfortable with the state of the product before release.
We also run RATs on every one of the RC releases, because they are candidates to be released. As such we treat all of them as the last one.
In the lifetime of a release we are constantly getting bugs reported by our users. Our community and users are the most important asset of Unity and we value this contribution more than anything else in the process of making a Unity version come to life. It is definitely not an easy task and we have posted multiple blog posts about how we handle this flow of bugs. Best place to read about it is The Great Incident Bash of 2015.
Throughout all our releases in the entire lifetime of the release we verify all bugs fixed. During verification we go through the steps to reproduce the bug, look around the immediate area around the fix to find fallout and then close it or reactivate it depending on the result.
During the lifetime of the engine, we also run our performance regression test suite multiple times. Sakari made a two part blog post about it two years ago and lots has happened since. It is a tough problem to solve consistently and we have invested a lot of effort into this, so we will make a part three to update you on what is going on these days.
It’s important to also understand that testing a platform is a separate discipline in itself. We are helped by having lots of our automation be able to run on all our platforms, but there is a lot of custom manual testing needed for each platform because they all have their independent functionality to consider.
Once a version has been shipped, it goes into sustained engineering (SE) mode. This team was founded two years ago and a lot has happened since. They are responsible for all the patches we send out and we will most certainly create an updated blogpost about how this team is now functioning.
I hope this sheds some light on the overall process of shipping a piece of software as complicated as Unity. We will post a lot more blogs about the details of all these parts. The automation we are doing is such a big part of our process and ability to ship all these platforms, so we will create a similar overview post dedicated to this.