How we share Android test suites with the world
In my last blog post I introduced you to our Runtime Test Framework and how it allows us to test the Unity Runtime API on a wide range of platforms. In this post I will tell you more about the test suite that we share with Google and others so they can integration test Unity with Android devices and the Android OS.
“What is our motivation for sharing this suite?” you might ask. It makes a lot of sense for us to push our test suites upstream and catch eventual problems as early as possible. We mitigate risk, in this case the risk that a certain device or a specific version of Android does not work with Unity. Also, the earlier we become aware of a problem the more time we will have to react and find a solution.
The Android Suite and Runner
The Unity Runtime Test Framework can generate a Unity project containing all tests, and then build an Android suite from it. We remove tests with external dependencies, e.g. WWW tests that require a running web server, but this is limited to a very small set of tests.
The external Android test runner is different from the one we use internally at Unity, which is based on C#/Mono. Our test frameworks and runners are very modular so replacing one test runner with another is very easy. The Android runner is based on the Android testing framework, and it starts a test suite by launching it as an activity on the device. As all runtime assertions are done player-side, on the Android device, the result of a test can be determined by the runner by using logcat and a simple parser.
The Runtime Test Framework can build an Android test suite in a few minutes, and running a pre-built suite takes 5-10 minutes on most devices (currently more than 700 tests covering all major areas of Unity as shown in the figure below).
Updating and releasing test suites
We typically update and ship a new version of the Android runtime test suite before we ship a new version of Unity. We like our bugs to be found and fixed as early as possible, and the results achieved so far are very promising. We have found bugs and we have fixed them before they did much harm. We are currently looking into finding more potential players who can benefit from this suite.
In my next blog post, which I will publish in a few weeks, I will tell you about our Graphics Testing Framework. Stay clear of bugs until then.