Unity Test Tools 1.2 have been released

March 12, 2014 in Asset Store, Technology

Hi there! My name is Tomek and I am the main developer behind Unity Test Tools. Ever since the first release we are actively developing the tools and trying to make them more robust, comprehensive and seamless for you. We wouldn’t be able to achieve that without your feedback so thank you very much for that!

Few days ago we released version 1.2 of Unity Test Tools. Today, I would like to highlight some of the recent changes and discuss main pain points you faced.

Assertion Component

Code stripping

We’ve been hearing from lots of people that the main issue that prevents them from using the assertion component is the lack of the option to strip the code out of the release builds. As we faced this issue, we came up with a solution, which is (duh!) very trivial.

We implemented scene post-process mechanism that removes all components from the built scenes. Nevertheless, the code of the assertion is still being included in the builds. This is due to Unity’s nature and the potency of managed languages (you still might want to use reflection on non-referenced code, don’t you?). But no worries! It’s not anything you can’t deal with.

The solution to completely strip the code from your build would be to implement your own build pipeline and build your project with BuildPipeline.BuildPlayer. See an example of custom build pipeline implementation below:

using UnityEngine;
using UnityEditor;

public class StripCodeAndBuild
{
   private static string assetsPath = "Assets/UnityTestTools";
   private static string buildTempPath = "Assets/Editor/UnityTestTools";

   [MenuItem ("Unity Test Tools/StripCodeAndBuild")]
   public static void StripCodeAndBuildStandalone ()
   {
      AssetDatabase.CreateFolder ("Assets""Editor");
      var result = AssetDatabase.MoveAsset (assetsPath, buildTempPath);

      if (string.IsNullOrEmpty (result))
      {
         BuildPipeline.BuildPlayer (new[] {EditorApplication.currentScene},
                                    @"C:\temp\release.exe",
                                    BuildTarget.StandaloneWindows,
                                    BuildOptions.None);
         result = AssetDatabase.MoveAsset (buildTempPath, assetsPath);
         if (string.IsNullOrEmpty (result))
            AssetDatabase.Refresh ();
         else
            Debug.LogWarning (result);
      }
      else
      {
         Debug.LogWarning (result);
      }
   }
}

The StripCodeAndBuildStandalone will move the whole UnityTestTools folder to a path under Editor folder, build a WindowsStandalone player, and then move the tools back to it’s original location. Any file under Editor folder is not included in the build.

We hope this will make some people happy and encourage them to use the assertion as a first barrier of defence against bugs!

Performance

As an assertion, the component should have as little performance impact on runtime as possible. We took a closer look and redesigned Assertion Component’s background code to make it run faster… almost 5 times faster! That is at least for a benchmark scene with 1000 check calls per frame.

Assertion Component overhead before optimization (~10ms per 1000 calls):

After optimization  (~2ms per 1000 calls):

Integration tests

Hierarchical structure

If you like nesting and grouping your test, this feature is for you! We made it possible to put one integration tests under another one (just like you would with GameObjects) to clear out the structure of tests. Once a test becomes a parent for another tests, it is treated as a test group. A test group can contain tests or more test groups. See the following screenshot:

 

The tests on the example scene were grouped by their result. Also the result triggers were placed accordingly. We hope this feature will make test scenes with large amount of tests more readable and easier to manage.

Platform Runner

Once you have your integration tests written, a natural thing to do is to run them on your the platform you are developing for. The Platform Runner was created to help you in automating this process. It will build selected scenes for the platform you choose and run it. Additionally, it allows you specify a path to save the results at.

A quick run option for running current scene is also available from the menu.

Reporting Results

Until now, the only way to get the results of a test run was to read the result file that was generated… if it was generated. Not every platform Unity can build for supports file system, and if they do, sometime it get tricky to get use of it (due to system permissions or accessing the file on the device).

To address that issue, we added a result report showed right on the screen of a device you are running the tests on. I will show you a list of failed tests and the logs that appeared while the test was running.

 

Unit Tests

Background runner

For your convenience we have rewritten the background runner mechanism to run unit tests even though the Unit Test Runner window is not focused. It will run seamlessly and notify you only if any of your unit tests failed.

Undo System

Instantiating GameObjects in a unit test is not a good idea in general, however, sometimes you might have no other choice (or no will to refactor your code!). The Unit Test Runner gives an option to run unit tests on a separate scene, which is a fast and easy way to clean the ‘pollution’  from the scene. This however gives some overhead of reopening your current scene after every run. To make unit tests execution fast and seamless you should avoid running them on a separate scene and use Unity’s undo system to backout any changes to the scene you make in your unit tests. Simply register them with Undo class or utilize the UnityUnitTest base class that comes with the tools.

Continuous Integration Systems accomodation

One of the most commonly asked questions from you are related to continuous integration systems. It’s quite obvious that once you have the automation written, you don’t want to run them manually every time but delegate this task to the machines. It’s called an automation for a reason.

To accommodate some of your requests we redesigned the batch runner for both unit tests and integration tests runners. We added possibility to specify result file path from as a command line argument and enabled integration tests to be run and build in a player from batch. See the documentation for changes.

 

Summary

Once again big hugs to all of you that contributed, shared their opinion and reported issues. There is a reason we released  the tools via the Asset Store and it’s all about you being involved in development. It allows us to rapidly respond for you feedback and accommodate your needs. We make the tools for you and we encourage you to contribute. Share your opinions and experience of using Unity Test Tools at Unity Forum or via Twitter (#UnityTestTools).

Along with working on the tools, we actively develop best practices and patterns for testing games. Stay tuned for a series of blog posts about that!

And remember: Don’t just make games, make them better!

 

Comments (12)

Subscribe to comments
  1. Lior Tal

    May 20, 2014 at 3:11 pm / 

    I have written a blog post briefly covering how to use this package: http://www.tallior.com/2014/05/20/introduction-to-unity-test-tools/

  2. Juergen Wind

    April 8, 2014 at 3:28 pm / 

    Hi Tomek,

    really nice testing tool.
    I have a bunch of standalone unit test assemblies that I would like to test in Unity.
    I found out that I can add them to “assembliesWithTests” in nUnitTestEngine.cs, but I thought there could be a nicer way.

    Keep up the good work !

  3. yoonmi hwang

    March 29, 2014 at 9:41 am / 

    to explore the real game,meet the friend , play with exciment

  4. Andrew Fray

    March 28, 2014 at 12:31 pm / 

    Hi Tomek, thanks for the testing tools.

    It’d be great to see Unity open-source the tools under MIT or something, allowing us to contribute back to the project and easily report issues (but with you keeping complete control of the project). It seems odd that this is free, yet I can’t easily share changes to it.

    Thanks!

  5. Elmy@Cheaperseeker

    March 20, 2014 at 4:01 am / 

    Stay tuned for a series of blog posts about that and expecting better performance.

  6. Tomek Paszek

    March 17, 2014 at 11:10 am / 

    @PETER DWYER
    Thanks!
    I also fixed the typo for your convenience. Have a good day! ;)

  7. Peter Dwyer

    March 17, 2014 at 10:58 am / 

    “We implemented scene post-process mechanism that removes all components from the builded scenes.”

    I struggle to get past this line without a schoolboy smirk please change it to

    “We implemented scene post-process mechanism that removes all components from the built scenes.”

    Great article by the way :-)

  8. Tomek Paszek

    March 16, 2014 at 1:04 pm / 

    Hi ZEPHYR ZHANG,
    It’s really great to hear such positive response! If you happen to attend Unite China I would love to have a chat with you. I’m giving there a talk about the tools.
    Tomek

  9. Zephyr Zhang

    March 16, 2014 at 7:35 am / 

    This is great news! we are using Unity Test Tools in our project for 2 months, maybe we are the first company in China to use it. We are refactoring the battle system, added hundreds of unit tests and dozens integration tests. This extension really makes TDD possible. We are very happy to see it keeping evolving. Thank you guys!

  10. Tomek Paszek

    March 14, 2014 at 10:55 am / 

    Hi LIOR TAL,
    No offence taken! If you know a better solution we would be glad to hear it. Other options to consider are:
    ConditionalAttribute along with defining a symbol for DEVELOPER/RELEASE builds. See Global Custom Defines section in Platform Dependent Compilation. This solution however won’t handle assets that are not scripts.
    Also, see the Build Player Pipeline section.

  11. Tyler Kirk

    March 14, 2014 at 2:13 am / 

    Very cool! Can’t wait to dig in!

    -T

  12. Lior Tal

    March 13, 2014 at 11:22 pm / 

    Thanks for the update.

    Not to offend anyone, but the first solution presented here smells like a bit of a hack (moving a folder back and forth).

Comments are closed.