Runtime Tests – Unity’s Runtime API Test Framework

June 2, 2013 in Tecnología

Hi, my name is Kasper Amstrup Andersen, and I am leading the Toolsmiths Team here at Unity. The Toolsmiths Team consists of 6 developers, and we work on tools, frameworks, and infrastructure – and then we also spend a great deal of time actually testing Unity. The team is located in Copenhagen (Denmark), Odessa (Ukraine), and Helsinki (Finland). In this blog post I will tell you about one of the frameworks we are currently working on: The Unity Runtime Test Framework.

At Unity QA our passion is “Quality”. We like testing and we love to be smart about the testing we do. We now have 4 teams of very dedicated and professional testers and developers that spend every day making sure that Unity “just works” for everyone. The teams are: Test Engineers, Test Developers, Toolsmiths, and Student Workers (we have previously written a bit about the teams). A huge challenge when testing Unity is actually breaking down the product in logical components where it can make sense to talk about test coverage. The main headache is Unity’s many platforms which are in many ways (but not all) orthogonal to the core feature set. To give you an example: We have areas and we have platforms, but then we also have platform-specific areas (and features). A testing matrix basically explodes.

In our Runtime API Test Framework we treat platforms as being completely orthogonal to features. This assumption allows us to make a specialized framework that gives a lot of coverage with very little effort. You write a test once; the framework will make it run everywhere.

Before kicking off, I’ll give you some numbers and quick facts:

  • Today, we have >750 unique Runtime API test cases.
  • More than 9000 Runtime API test cases (across more than 10 platforms/players) are run in a single automated build verification test suite. The number is rapidly increasing.
  • We easily execute more than 150.000 Runtime API test cases per day.
  • The framework runs on all of Unity’s supported platforms/players.
  • The framework (and our build system) runs tests on multiple platforms in parallel. Average execution time for the entire suite is less than 30 minutes, and half that time is spent setting up the environment and building players (actual test execution time is about 15 minutes on average).
  • The framework is also used by external companies to integration test their products with Unity players.

The Runtime Test Framework

We have named our Runtime API Test Framework “Runtime Tests”. So, what is a Runtime Test? It’s basically a test case that verifies some functionality in the Unity Runtime Classes. The Runtime Test Framework allows us to write a test case in C# and then it will run on all platforms without any boiler plating code. The framework is integrated into our build system and developers and testers run all tests in the branches where they are working + in Trunk, our shared code repository, whenever there are changes. Some of the framework’s core features are:

  • Write tests once. The framework will make them run everywhere.
  • A library of player-side assertion methods that are supported on all players.
  • Tests are fast (execution time ranges from milliseconds to a few seconds).
  • Tests run in scene isolation. Each test is executed in a separate scene.
  • Tests can easily be disabled, also on specific platforms only.
  • The framework is connected to a powerful back-end where we have close to real-time monitoring of key metrics.
  • Automation for new players can be added basically without modifying the core framework.

The Runtime Test Framework consists of the following modules:

  • Unity.Automation: Functionality for starting and interacting with the Unity Editor.
  • Unity.Automation.Players: Functionality for starting and interacting with players and for using the build pipeline.
  • Unity.RuntimeTests.Framework: The core framework logic.
  • Unity.RuntimeTests.Runner: The test runner. Will load the module that contains test cases and start test execution.
  • Unity.RuntimeTests: Runtime API test cases.

The different modules, and how they depend on each other, is shown in the figure below.

Runtime Test Framework modules and dependencies.

The framework consists of runner-side part and a player-side part. The player-side uses a very barebones implementation (no use of interfaces, generics etc.) since it must run on all players (also players we might consider to add in the future).

An example of actual test execution is shown in the figure below.

Runtime Test execution.

The runner will load the test assembly (Unity.RuntimeTests) and then pass control to the framework (1, 2). The framework will start a Unity Editor process (3). Then, it will create (or copy in) all assets required to run tests, and start building a scene for each test (4). This is done by invoking saved delegates inside the Editor process. Pre-made assets are stored together with the test assembly. For tests using the WWW class the framework will start a webserver where [request, response] pairs can be saved during setup (5). At runtime a test can then make a www request which will be redirected to the webserver by the framework. Finally, a player is built and started, test results are consumed, and a test report will be generated (6, 7a, 7b, 8).

The reporting backend

The reporting backend scans the test assembly (Unity.RuntimeTests) and monitors all tests. This allows us to track various metrics. An example is shown below.

Runtime Tests grouped by area.

This chart shows all Runtime Tests grouped by area. The horizontal axis shows “area” and the vertical axis shows “number of tests”.

That’s it…

So, there you have it. That’s our Runtime Tests. We like them and they make writing tests for all Unity players easy and fun. And then they simply give a lot of “test-bang-for-the-buck”. But this is just one of the many things we’re currently working on in the Toolsmiths Team. I look forward to share more in future blog posts.

Comments (12)

Subscribe to comments
  1. Osher

    July 23, 2013 at 1:28 pm / 

    +1 for opening the test framework.
    +1 for what AJ said
    +1 for Jerome as well, but that can be added later.
    The sooner you have inherent testing standards – the better

  2. AJ

    June 21, 2013 at 8:13 am / 

    As someone currently shopping for a game engine and also a huge advocate for automation testing, sharing your internal testing framework would prove valuable not only to existing Unity users, but also to prospective users of your engine. If it’s easier to write and maintain automated tests than anything else out there, you’ll have a huge leg up on any competition.

  3. Gary

    June 12, 2013 at 7:47 am / 

    I agree with Mike and the others — this would be fantastic to have accessible to use on the products of Unity3D. A built in testing / integration framework could serve to vastly improve the quality of the games produced with Unity. Even as a Pro feature or as some style of add-on (like the Asset Server), I’d be chomping at the bit to use it locally.

  4. Jerome Lacoste

    June 11, 2013 at 1:21 am / 

    When opening it, consider making it possible for us to create bug reports using the framework. Given the proper examples and guidelines it could make bug reporting/fixing more efficient.

  5. Marc 'Dreamora' Schärer

    June 10, 2013 at 5:48 am / 

    Would absolutely love to give this framework a go.

    Testing in the past was either near impossible, required many extra loops (external libraries to run tests in MD or VS) or required the usage of Test Star (http://u3d.as/content/eye3ware/test-star/2dB) which on its own is a hit or miss experience (yet pretty nice for simulated interactive tests) and I would like to adopt something that might help us get around this in a ‘standardized’ and platform scalable way.

  6. Natali Abbortini

    June 5, 2013 at 6:20 am / 

    Will Unity 3D soon support realtime global illumination using voxel cone tracing ?

  7. Erik

    June 3, 2013 at 2:15 pm / 

    Please do share ! Would be great to have a solid way to test our own code like that !

  8. Ashkan

    June 3, 2013 at 10:36 am / 

    Really nice info.
    As some others said, currently doing unit tests and TDD in unity is not that easy.
    Since we are smaller teams we should write lots of scaffolding code for some of our tests. If some of that could become easier it would be great.

    Having something as easy as visual studio 2012 and TFS provides for windows developers is not possible for us in foreseeable future but making it somehow easier would be great.

  9. Kasper Andersen

    June 3, 2013 at 4:41 am / 

    Hi, I’m glad you like the post.

    The framework is still work-in-progress, but I don’t see why we can’t share what we have with everyone. We don’t have any concrete plans yet, and I can’t promise anything, but I’ll keep everyone posted when I know more.

  10. Martijn Zandvliet

    June 3, 2013 at 2:31 am / 

    I would also like to see something like this framework being publicly released. I haven’t bothered much with TDD yet in Unity, and this would probably tip me over.

  11. Lior Tal

    June 2, 2013 at 10:07 pm / 

    Thanks for the post!
    Any chances at least some of the framework is going to be exposed to the public?

  12. Chen

    June 2, 2013 at 9:08 pm / 

    Smart QA – using Unity feels the tremendous effort put into testing it.
    Keep up the solid work guys!

Comments are closed.