Categories & Tags
Archive

Unity Test Tools Released

December 18, 2013 in Tech by

Through the past 2 years Unity QA has expanded and built tools, frameworks, and test rigs for internal use, something we have previously blogged about. Through all this work we have done, we have created a lot of value internally in Unity and we want to give our users access to these awesome tools. Today we have released version one of the Unity Test Tools on the Asset Store. Get it here:

Thus we made the decision to make Unity Test Tools available for our users, which we hope will help you attain a high quality in your code while developing your games.

Unit Test

The lowest and most efficient level to do automation is at the unit level. We have decided to use nUnit as the basis for our unit test framework, which is a well known framework for those already writing unit tests.

Unit_Test_Runner

We have ensured that the integration to the editor is intuitive and simple, with the option of having automatic execution on every recompile, so you have immediate feedback while you write your code. Another important aspect of a test framework is the ability to make a build pipeline where unit tests are executed without a head and the Unity Test Tools give you this option as well.

Integration Test

In order for you to test the integration between components, objects and assets, you need a higher level of tests and for this we have made the Integration Test Framework. Integration tests are also the easiest way of starting to write automation tests on a game in development, since they don’t have requirements on the architecture of the code.

Integration_Test_Runner

The simplest use of these would be to make a scene, use assets and gameobjects, and set up the conditions you want to verify. The execution is as simple as with the unit tests, where the execution will cycle through the scenes containing tests and execute each test for you. This framework can also be integrated into a build pipeline seamlessly, so you can test all levels of your project from commandline.

Assertion Component

The assertion component is able to monitor the state of gameobjects while in playmode and pause the game if the condition is met. The primary use is debugging, where you can make the game stop in the exact frame the condition is met, thus enabling you to see the entire picture of when it occurs.

To help manage the assertion components, we have added an explorer which is similar to a list of breakpoints in code, so you have an overview of the states and where the components are placed. The component can evaluate complex conditions runtime and thus is more powerful than a normal breakpoint.

Assertion_Overview

In order to enable the build and release pipeline, we have made it possible for you to disable assertion components in release builds, thus making sure you don’t hit them in a released game.

Examples

In the package you download you will find a full set of examples and a comprehensive documentation on how each of the tools work.

The Future

Releasing the tools is just the beginning for us. We will be committed to release tutorials and patterns which will help you structure your projects such that they are testable. We will also continue improving on the tools and increase integration into Unity, all with the aim of making it easy for you to start testing your projects.

Happy testing!

Share this post

Comments (40)

Comments are closed.

Gus
18 Dec 2013, 4:08 pm

Very good news! Will try it asap. Thanks!

18 Dec 2013, 4:12 pm

Thanks for releasing this great tool to the public. Your game development tools have helped my company thrive in the online gaming industry and I have nothing but praise for anything you guys put out. Thanks for being so awesome!

18 Dec 2013, 4:22 pm

Thanks for releasing this wonderful tools

18 Dec 2013, 4:31 pm

This is fantastic news, thanks for this!

18 Dec 2013, 5:30 pm

Awesome news!
This tools will be useful for pre-QA testing.

Thanks Unity!

18 Dec 2013, 5:34 pm

Nice!!
Maybe the most useful thing in the last year for many of us, specially those who do more complex testable stuff like AI and online games.

Gerry
18 Dec 2013, 7:06 pm

Terrific, thank you!

Juan Fornos
18 Dec 2013, 7:11 pm

Is it possible to run the tests via the command line (I’m thinking in continuous integration).

Juan Fornos
18 Dec 2013, 7:15 pm

Sorry, too anxious. Already stated in the article…

Dogukan
18 Dec 2013, 9:52 pm

Wow, thanks for sharing such powerful tool to us and it’s for free!

18 Dec 2013, 10:17 pm

Wow. Thanks for sharing. This is so cool.

Matt
18 Dec 2013, 11:04 pm

Brilliant, thanks Unity folks!

ZimM
19 Dec 2013, 3:33 am

What about running tests on the build platforms? It’d be very useful for iOS, as there’s a lot of AOT stuff that’s hardly predictable, even when everything is fine in editor,

19 Dec 2013, 5:42 am

<3

19 Dec 2013, 5:52 am

Great! This are the tools we were waiting for :-)

Tomek
19 Dec 2013, 8:17 am

@ZIMM
You can build integration tests for some platforms. We still experiment with different platforms and we didn’t verify running test on iOS yet. You can build and run integration test in standalone, webplayer and android.

19 Dec 2013, 8:32 am

Does this mean we will finally get to see that most awesome draw call inspector from the ninja bootcamp? I’ve been waiting for that one since the first time I saw it! :D

Thomas Petersen
19 Dec 2013, 9:03 am

@JACCO: Uhm, what? This has nothing to do with the draw call inspector.

19 Dec 2013, 3:31 pm

Any example of how to test GUI related stuff :
I set mouse at x:200, y:100 and Click.
I test if my button is pressed

Thomas Petersen
19 Dec 2013, 4:59 pm

@martijn: the question is what you attempt to test? In this specific case, you are not really interested in testing the coordinate on the screen, because that is of little interest and would be very fragile. Instead, you are interested in the change of underlying state change in the game which happens as a result of the subsequent call. Separating that behaviour from the GUI presentation is vital if you want to achieve a testable application.

We intend to release more tutorials and patterns on achieving testability in a Unity project and what you requested here is a good example of what we could write about.

20 Dec 2013, 1:14 am

Hi, love that you’ve released these tools, and from what I can see they look really helpful, however I’m having a problem with some of the unit testing.

In the docs, under the NUnit section, you say:

“For managing GameObjects on the scene you can use the UnityTest class which provides you with a method for creating GameObjects and does the cleanup automatically”

As far as I can tell, there is no such UnityTest class, only a namespace, and that namespace neither contains a UnityTest class nor anything that is named in such a way to suggest it will safely create GameObjects during a test.

I’m probably just blindly missing something obvious, but a helping hand (and docs fix) would be great.

Robotron18
20 Dec 2013, 4:56 am

“You will see the sphere falling down and once it falls below the cube, the editor should pause.”

AssertionExampleScene in Examples. Not paused editor, only exeption message in console.

Tomek Paszek
20 Dec 2013, 10:41 am

@NICK UDELL
The class the documentation is referring to has been lost when we published the package. It will be fixed with the next update.
FYI it’s just a simple class that provides you a method to create GOs and keeps track of them. The purpose it to remove all GOs created by the test once it has finished.

@ROBOTRON18
It’s a documentation bug. The editor will pause when an exception is thrown (assertion failed) when you check the ‘Error pause’ checkbox in the console window.

Tomek Paszek
20 Dec 2013, 11:16 am

@NICK UDELL
For now you can get the missing file here: http://forum.unity3d.com/threads/218659-Unity-Test-Tools-missing-class-file?p=1459615

20 Dec 2013, 2:44 pm

@Tomek thanks so much, that’s excellent!

Zephyr Zhang
21 Dec 2013, 7:41 am

I tried to use TDD in Unity for 2 years, always fails. Today Finally! Thank you Unity!

David Mitchell
21 Dec 2013, 6:11 pm

Unity never stops impressing me. This is fantastic! Thanks, Unity.

koblavi
24 Dec 2013, 10:04 am

Some of the most committed developers I know! This is awesome. I’m yet to try it.You should go ahead and integrate it fully in 4.4/4.5

Thomas Petersen
24 Dec 2013, 12:07 pm

@Koblavi: We have plans to integrate them fully, but when it will happen I don’t know.

26 Dec 2013, 4:15 pm

Thanks for releasing this wonderful tools

Joseph Osborn
26 Dec 2013, 5:52 pm

What’s the source code license for the Unity Test Tools? Its components seem to be BSD and MIT licensed, but what about the original code in the tools? Is it public domain?

Thomas Petersen
26 Dec 2013, 6:01 pm

@joseph:the license for the referenced tools such as n Unit obviously follows those tools as mentioned in the package. Our code is completely free for use anywhere without mention. We want to enable everyone to use them to improve code quality without impediments.

Steve
29 Dec 2013, 6:22 am

Nice! I never really had a chance to really get into unit testing. I had one job that I started working with it, then the company went belly up. I am very happy I can now lean!

Just one question though. I get the basic Unit Testing and the Integration Testing. But I have a base script that deals with a lot of logic based on TIme.time. It seems I can’t use the basic unit tests with asserts seeing as Time.time is meaningless while the tests run. If I use Thread.Sleep(t) it just delays the Time as well. It’s doesn’t inherit Misbehavior either.

So how do I test these methods, without wrapping the class? As that would be messy would it not?

Thanks in advance for any input.

Thomas Petersen
29 Dec 2013, 11:33 am

@steve: excellent question, more of which I hope you will put on our forums in the future. To answer you now, having a unittest in any way be dependent on time, be it in Unity or elsewhere, is a bad idea. The test will be very fragile and you can’t control the output. Usually you would make a mock of Time (wrap it), but in this particular case I believe you would be able to use an integration test better, because it plays in a regular scene.

Steve
29 Dec 2013, 7:33 pm

@Thomas, I see. Thank you for your insight. I was asking on here because it seems Unity Test Tools are still pretty fresh to the community. This is the most likely area to get someone that would be a bit more knowledgeable. You are correct though. How will it become popular on the forums if no one posts there right?

Thanks again.

Thomas Petersen
29 Dec 2013, 7:38 pm

@Steve: Exactly :-) And well, if enough questions about how to test pops up, we might create a subforum for just this purpose. It was a good question, so we would be happy to answer more of those.

Steve
30 Dec 2013, 9:16 pm

@Thomas: I also just had a thought. Not sure if this would be viable or not (as I stated before I am still pretty new to TDD, but would making a Time wrapper class that allows you to simulate and control time be better than wrapping it and using it in an integration test? I realize this would most likely depend on the depth of Time use?

Anyway was just a though. Figured I would ask before I start investing too much time on a whim.

Thomas Petersen
30 Dec 2013, 9:27 pm

@Steve: It’s not as much TDD as it is about automation. Your question depends on how well isolated your system under test (SUT) is. If you are only dependent on Time and don’t have other dependencies in your SUT which_might_ use Time, you can wrap it (look up how to mock). That is preferable to do on new code, since unittests are way faster than other types of test, but can be hard on an existing codebase.

And once again, this really needs to be on the forums… :-)

Steve
30 Dec 2013, 9:35 pm

@Thomas: I apologize for continuing it here. My reasoning behind it was for people that may be looking for a similar question, will find the entire conversation here apposed to it being segmented between here and on a forum. I suppose I should have posted a new question in the forums and just send a link here. I didn’t think that far ahead. I assure you any other inquiries will be in the forums.

Again thanks for all your insight.

Nathan
1 Jan 2014, 6:56 pm

Thanks Thomas and whoever else worked on this. This has been a much needed system in Unity for a while now. I’m really excited to dive in and test out the testing tools you all have put together:)

Leave a Reply

Comments are closed.