Unity Test Tools Released

December 18, 2013 in Technology

Through the past 2 years Unity QA has expanded and built tools, frameworks, and test rigs for internal use, something we have previously blogged about. Through all this work we have done, we have created a lot of value internally in Unity and we want to give our users access to these awesome tools. Today we have released version one of the Unity Test Tools on the Asset Store. Get it here:

Thus we made the decision to make Unity Test Tools available for our users, which we hope will help you attain a high quality in your code while developing your games.

Unit Test

The lowest and most efficient level to do automation is at the unit level. We have decided to use nUnit as the basis for our unit test framework, which is a well known framework for those already writing unit tests.

Unit_Test_Runner

We have ensured that the integration to the editor is intuitive and simple, with the option of having automatic execution on every recompile, so you have immediate feedback while you write your code. Another important aspect of a test framework is the ability to make a build pipeline where unit tests are executed without a head and the Unity Test Tools give you this option as well.

Integration Test

In order for you to test the integration between components, objects and assets, you need a higher level of tests and for this we have made the Integration Test Framework. Integration tests are also the easiest way of starting to write automation tests on a game in development, since they don’t have requirements on the architecture of the code.

Integration_Test_Runner

The simplest use of these would be to make a scene, use assets and gameobjects, and set up the conditions you want to verify. The execution is as simple as with the unit tests, where the execution will cycle through the scenes containing tests and execute each test for you. This framework can also be integrated into a build pipeline seamlessly, so you can test all levels of your project from commandline.

Assertion Component

The assertion component is able to monitor the state of gameobjects while in playmode and pause the game if the condition is met. The primary use is debugging, where you can make the game stop in the exact frame the condition is met, thus enabling you to see the entire picture of when it occurs.

To help manage the assertion components, we have added an explorer which is similar to a list of breakpoints in code, so you have an overview of the states and where the components are placed. The component can evaluate complex conditions runtime and thus is more powerful than a normal breakpoint.

Assertion_Overview

In order to enable the build and release pipeline, we have made it possible for you to disable assertion components in release builds, thus making sure you don’t hit them in a released game.

Examples

In the package you download you will find a full set of examples and a comprehensive documentation on how each of the tools work.

The Future

Releasing the tools is just the beginning for us. We will be committed to release tutorials and patterns which will help you structure your projects such that they are testable. We will also continue improving on the tools and increase integration into Unity, all with the aim of making it easy for you to start testing your projects.

Happy testing!

Comments (40)

Subscribe to comments
  1. Nathan

    January 1, 2014 at 6:56 pm / 

    Thanks Thomas and whoever else worked on this. This has been a much needed system in Unity for a while now. I’m really excited to dive in and test out the testing tools you all have put together:)

  2. Steve

    December 30, 2013 at 9:35 pm / 

    @Thomas: I apologize for continuing it here. My reasoning behind it was for people that may be looking for a similar question, will find the entire conversation here apposed to it being segmented between here and on a forum. I suppose I should have posted a new question in the forums and just send a link here. I didn’t think that far ahead. I assure you any other inquiries will be in the forums.

    Again thanks for all your insight.

  3. Thomas Petersen

    December 30, 2013 at 9:27 pm / 

    @Steve: It’s not as much TDD as it is about automation. Your question depends on how well isolated your system under test (SUT) is. If you are only dependent on Time and don’t have other dependencies in your SUT which_might_ use Time, you can wrap it (look up how to mock). That is preferable to do on new code, since unittests are way faster than other types of test, but can be hard on an existing codebase.

    And once again, this really needs to be on the forums… :-)

  4. Steve

    December 30, 2013 at 9:16 pm / 

    @Thomas: I also just had a thought. Not sure if this would be viable or not (as I stated before I am still pretty new to TDD, but would making a Time wrapper class that allows you to simulate and control time be better than wrapping it and using it in an integration test? I realize this would most likely depend on the depth of Time use?

    Anyway was just a though. Figured I would ask before I start investing too much time on a whim.

  5. Thomas Petersen

    December 29, 2013 at 7:38 pm / 

    @Steve: Exactly :-) And well, if enough questions about how to test pops up, we might create a subforum for just this purpose. It was a good question, so we would be happy to answer more of those.

  6. Steve

    December 29, 2013 at 7:33 pm / 

    @Thomas, I see. Thank you for your insight. I was asking on here because it seems Unity Test Tools are still pretty fresh to the community. This is the most likely area to get someone that would be a bit more knowledgeable. You are correct though. How will it become popular on the forums if no one posts there right?

    Thanks again.

  7. Thomas Petersen

    December 29, 2013 at 11:33 am / 

    @steve: excellent question, more of which I hope you will put on our forums in the future. To answer you now, having a unittest in any way be dependent on time, be it in Unity or elsewhere, is a bad idea. The test will be very fragile and you can’t control the output. Usually you would make a mock of Time (wrap it), but in this particular case I believe you would be able to use an integration test better, because it plays in a regular scene.

  8. Steve

    December 29, 2013 at 6:22 am / 

    Nice! I never really had a chance to really get into unit testing. I had one job that I started working with it, then the company went belly up. I am very happy I can now lean!

    Just one question though. I get the basic Unit Testing and the Integration Testing. But I have a base script that deals with a lot of logic based on TIme.time. It seems I can’t use the basic unit tests with asserts seeing as Time.time is meaningless while the tests run. If I use Thread.Sleep(t) it just delays the Time as well. It’s doesn’t inherit Misbehavior either.

    So how do I test these methods, without wrapping the class? As that would be messy would it not?

    Thanks in advance for any input.

  9. Thomas Petersen

    December 26, 2013 at 6:01 pm / 

    @joseph:the license for the referenced tools such as n Unit obviously follows those tools as mentioned in the package. Our code is completely free for use anywhere without mention. We want to enable everyone to use them to improve code quality without impediments.

  10. Joseph Osborn

    December 26, 2013 at 5:52 pm / 

    What’s the source code license for the Unity Test Tools? Its components seem to be BSD and MIT licensed, but what about the original code in the tools? Is it public domain?

  11. arabchat

    December 26, 2013 at 4:15 pm / 

    Thanks for releasing this wonderful tools

  12. Thomas Petersen

    December 24, 2013 at 12:07 pm / 

    @Koblavi: We have plans to integrate them fully, but when it will happen I don’t know.

  13. koblavi

    December 24, 2013 at 10:04 am / 

    Some of the most committed developers I know! This is awesome. I’m yet to try it.You should go ahead and integrate it fully in 4.4/4.5

  14. David Mitchell

    December 21, 2013 at 6:11 pm / 

    Unity never stops impressing me. This is fantastic! Thanks, Unity.

  15. Zephyr Zhang

    December 21, 2013 at 7:41 am / 

    I tried to use TDD in Unity for 2 years, always fails. Today Finally! Thank you Unity!

  16. Nick Udell

    December 20, 2013 at 2:44 pm / 

    @Tomek thanks so much, that’s excellent!

  17. Tomek Paszek

    December 20, 2013 at 11:16 am / 

    @NICK UDELL
    For now you can get the missing file here: http://forum.unity3d.com/threads/218659-Unity-Test-Tools-missing-class-file?p=1459615

  18. Tomek Paszek

    December 20, 2013 at 10:41 am / 

    @NICK UDELL
    The class the documentation is referring to has been lost when we published the package. It will be fixed with the next update.
    FYI it’s just a simple class that provides you a method to create GOs and keeps track of them. The purpose it to remove all GOs created by the test once it has finished.

    @ROBOTRON18
    It’s a documentation bug. The editor will pause when an exception is thrown (assertion failed) when you check the ‘Error pause’ checkbox in the console window.

  19. Robotron18

    December 20, 2013 at 4:56 am / 

    “You will see the sphere falling down and once it falls below the cube, the editor should pause.”

    AssertionExampleScene in Examples. Not paused editor, only exeption message in console.

  20. Nick Udell

    December 20, 2013 at 1:14 am / 

    Hi, love that you’ve released these tools, and from what I can see they look really helpful, however I’m having a problem with some of the unit testing.

    In the docs, under the NUnit section, you say:

    “For managing GameObjects on the scene you can use the UnityTest class which provides you with a method for creating GameObjects and does the cleanup automatically”

    As far as I can tell, there is no such UnityTest class, only a namespace, and that namespace neither contains a UnityTest class nor anything that is named in such a way to suggest it will safely create GameObjects during a test.

    I’m probably just blindly missing something obvious, but a helping hand (and docs fix) would be great.

  21. Thomas Petersen

    December 19, 2013 at 4:59 pm / 

    @martijn: the question is what you attempt to test? In this specific case, you are not really interested in testing the coordinate on the screen, because that is of little interest and would be very fragile. Instead, you are interested in the change of underlying state change in the game which happens as a result of the subsequent call. Separating that behaviour from the GUI presentation is vital if you want to achieve a testable application.

    We intend to release more tutorials and patterns on achieving testability in a Unity project and what you requested here is a good example of what we could write about.

  22. Martijn

    December 19, 2013 at 3:31 pm / 

    Any example of how to test GUI related stuff :
    I set mouse at x:200, y:100 and Click.
    I test if my button is pressed

  23. Thomas Petersen

    December 19, 2013 at 9:03 am / 

    @JACCO: Uhm, what? This has nothing to do with the draw call inspector.

  24. Jacco

    December 19, 2013 at 8:32 am / 

    Does this mean we will finally get to see that most awesome draw call inspector from the ninja bootcamp? I’ve been waiting for that one since the first time I saw it! :D

  25. Tomek

    December 19, 2013 at 8:17 am / 

    @ZIMM
    You can build integration tests for some platforms. We still experiment with different platforms and we didn’t verify running test on iOS yet. You can build and run integration test in standalone, webplayer and android.

  26. Apol3d

    December 19, 2013 at 5:52 am / 

    Great! This are the tools we were waiting for :-)

  27. kormyen

    December 19, 2013 at 5:42 am / 

    <3

  28. ZimM

    December 19, 2013 at 3:33 am / 

    What about running tests on the build platforms? It’d be very useful for iOS, as there’s a lot of AOT stuff that’s hardly predictable, even when everything is fine in editor,

  29. Matt

    December 18, 2013 at 11:04 pm / 

    Brilliant, thanks Unity folks!

  30. hdsenevi

    December 18, 2013 at 10:17 pm / 

    Wow. Thanks for sharing. This is so cool.

  31. Dogukan

    December 18, 2013 at 9:52 pm / 

    Wow, thanks for sharing such powerful tool to us and it’s for free!

  32. Juan Fornos

    December 18, 2013 at 7:15 pm / 

    Sorry, too anxious. Already stated in the article…

  33. Juan Fornos

    December 18, 2013 at 7:11 pm / 

    Is it possible to run the tests via the command line (I’m thinking in continuous integration).

  34. Gerry

    December 18, 2013 at 7:06 pm / 

    Terrific, thank you!

  35. Ashkan

    December 18, 2013 at 5:34 pm / 

    Nice!!
    Maybe the most useful thing in the last year for many of us, specially those who do more complex testable stuff like AI and online games.

  36. Kenli

    December 18, 2013 at 5:30 pm / 

    Awesome news!
    This tools will be useful for pre-QA testing.

    Thanks Unity!

  37. Anthony Thomas

    December 18, 2013 at 4:31 pm / 

    This is fantastic news, thanks for this!

  38. Avinash

    December 18, 2013 at 4:22 pm / 

    Thanks for releasing this wonderful tools

  39. Mike Wojcik

    December 18, 2013 at 4:12 pm / 

    Thanks for releasing this great tool to the public. Your game development tools have helped my company thrive in the online gaming industry and I have nothing but praise for anything you guys put out. Thanks for being so awesome!

  40. Gus

    December 18, 2013 at 4:08 pm / 

    Very good news! Will try it asap. Thanks!

Comments are closed.