Search Unity

When I encounter an API in Unity that I am unfamiliar with the first thing I (and most of us) do is go to the Unity Scripting API Manual to see how it works via one of the examples. If that example will not compile when I try it, I assume that I must be doing something wrong. The example couldn’t possibly be broken, could it…?

This is how I discovered that we do indeed have examples in our scripting docs that do not compile, as a result of API changes over time and the odd case of an the example never compiling to start with. At Unity we have a lot of freedom in how we work; if we see a problem we can report it to the relevant team or fix it ourselves. At one of our recent Sustained Engineering team weeks we decided to do our own hackweek and picked several issues we wanted to tackle. Some of us chose to look into a solution for there being broken examples in the scripting docs.

There are about 15,000 scripting docs pages. Not all of them contain examples (a different problem which we are working to improve); however a large portion do. Going through each example and testing them manually would be unachievable in a week. It would not solve the problem of API changes or broken examples being written in the future either either.

Last year as part of the Unity 5.3 release we included a new feature called the Editor Test Runner. This is a unit test framework that can be run from within Unity. We have been using the Editor Test Runner internally for our own automated tests since its introduction. I decided to tackle the problem using an editor test. All our scripting docs are stored in XML files which we edit through an internal Unity project.

The code to parse all these files is already available in this project so it made sense to add the editor test into the same project so we could reuse it.

In our editor test framework (which is using NUnit) there is an attribute that can be applied to a test called TestCaseSource. This lets a test be run multiple times with different source data. In this case the source data would be our list of script examples.

Using this method now shows a list of all the tests that will be run in the test runner. Each test can be run individually or they can all be run using the Run All option.

To compile the examples we use CodeDomProvider. It allows us to pass in one or more strings that represent a script, and it will compile and return information on errors and warnings.

This is a cutdown version (XML parsing removed) of the first iteration of the test:

And it worked! We needed to make some small changes in how we compile the examples, though, as some scripts are designed to go together as a larger example. To check for this we compiled them separately; if we found an error, we then compiled them again combined to see if that worked.

Some examples are written as single lines of code which are not wrapped in a class or function. We could fix this by wrapping them in our test, but we have a rule that all examples should compile standalone (i.e. if a user copies and pastes it into a new file it should compile and work), so we count those examples as test failures.

The test was now in a state where it could be run as part of our build verification on the path to trunk. However there was one small problem: the test took 30 minutes to run. This is far too long for a test running in build verification, considering we run around 7000 builds a day.

The test was running sequentially, one script after another, but there was no reason we could not run them in parallel as the tests were independent of each other and did not need to make any calls to the Unity API;and we are only testing that they compile, not the behaviour. Introducing ThreadPool, a .NET API that can be used to execute tasks in parallel. We push the tests as individual tasks into the ThreadPool and they will be executed as soon as a thread becomes available. This needs to be driven from a single function, meaning that we can’t have individual NUnit test cases for testing specific examples from the docs. As a result we lose the ability to run any one of the tests individually, but we gain the ability to run them all quickly.

This took the test time from 30 minutes to 2, which is fine for running as part of our build verification.

Since we couldn’t test individual examples with NUnit any more, we added a button to the scripting doc editor to allow developers to test the examples as they write them. The script with an error is now colored red when the test is run and error messages are displayed beneath.

When the test was first run we had 326 failures which I whitelisted (so they could be fixed at a later date). We now have that down to 32, of which most are failures in the test runner mainly due to not having access to some specific assemblies. There have been no new issues introduced and we can rest assured that when we deprecate parts of the API the test will fail and we can then update the example to use the new API.

Overall I thought this was an interesting use of the Editor Test Runner. It does have some limitations: We only test C# examples, and I have not managed to get JS compilation working, although that won’t be an issue in the future.

Here is the full test.

18 replies on “Verifying the scripting docs – Fun with EditorTests”

What good is it to run 7000 builds per day, considering the typical work day of 8 hours, which is 480 minutes? At my work place we run a build each time someone pushes a commit, which is good enough for us.

Why didn’t you use “NUnit.Framework.ParallelizableAttribute”? Or am i misunderstanding how this attribute works?

Nice one!
However there’s probably a typo:
At the part about performance, you first say it took 30 minutes to run, and later you say that multithreading reduced the time needed from 20 to 2.
Is it 20 or 30 minutes?

Just wondering when you guys at Unity are going to conform to proper c# coding standards? Your c# examples are littered with incorrect c# language notation, and the code you show off in this blog post does not look like something created by a real c# programmer. Oh yes, it works, but it looks like sh#t :)

const string k_PathToApiDocs should be const string pathToApiDocs. There is no such thing as a lowercaseletterunderscorecamelcase in the c# coding standards. Have a look for yourself, please. I highly recommend that you install ReSharper on all your dev machines and start getting some nice coding standard warnings and refactoring help.

In my opinion, your comment is full of sh#t.

People are not obliged to follow YOUR coding convention, even if it is widely used.

Understand my point that I do agree that Unity’s naming is ugly, but I won’t complain because it is different than mine.

Comments are closed.