Unit testing at the speed of light with Unity Test Tools

July 28, 2014 in 技术

It’s time to tell a little bit more about NSubstitute library that ships with our Unity Test Tools and patterns of it’s effective usage.

Each software system consists of units of functionality. In object oriented languages, the minimal unit of functionality is a method. These methods usually depend on other methods and classes. If you have to test a method, some challenges will arise.

  • First challenge is that external dependencies are not easy to set up, e.g. some objects with complex initialization might be required.
  • Second challenge is that the test verifies specific execution path that requires certain behavior from other classes that are used.
  • Finally, calling methods of external classes might lead to some changes in an environment that could not be rolled back, e.g. deleting a real record from a database.

Unit test is about testing unit of functionality in an isolated environment. Isolated means with all dependencies mocked up. Which means that test works in a test specific environment where only one execution path is possible.

Test doubles substitute real dependencies of unit under test, forming test specific environment and making unit tests fast and robust.

There are five test double patterns: Dummy object, Test stub, Test spy, Mock and Fake.

Dummy object

This article uses a simple Space Trader game clone to demonstrate the usage of test doubles and the NSubstitute library. This game is centred around the spaceship controlled by the player.

The player can equip weapons to fight pirates. The spaceship has weapon slots and a weapon can be equipped if an empty weapon slot is available.

Test scenario: make sure that weapon slot is occupied when weapon is equipped.

  • Take a spaceship with one free weapon slot
  • Take any weapon
  • Equip the weapon
  • Check that no weapon slots are available

unit-testing-at-the-speed-of-light-with-unity-test-tools-image15

Test with dummy object created manually Test with dummy object created manually.

Test with NSubstitute based dummy object Test with NSubstitute based dummy object

Weapon object is nominal in this scenario which means that its methods and properties are not intended to be used in this execution path. It is just a required parameter and anything that implements IWeapon interface can be used, including null when method has no null parameter check. This object usage is called dummy object pattern.

There are two approaches to create dummy objects.

The first approach is to create dummy object manually. It makes sense to have only one dummy object per real object/interface and IDE functions will help you to generate classes that implement interface. Creating dummy objects this way and storing them with your tests is not a big deal.

unit-testing-at-the-speed-of-light-with-unity-test-tools-image10

The second approach is to use NSubstitute that is shipped with Unity Test Tools:
unit-testing-at-the-speed-of-light-with-unity-test-tools-image07

Remember that passing null as a parameter when method has no null parameter check is also considered to be dummy object pattern. Null is not descriptive though.

All described methods are valid and very easy to use.

Test Stub

But what if spaceship’s method should return some value and this value is taken from weapon object? In this case, weapon is not a Dummy object anymore. It’s time for a test stub.

Test stub returns values for testing one specific execution path. E.g. BrokenWeaponStub, IncompatibleWeaponStub and others that let you test specific scenarios.

Test scenario: make sure that ship shoots at least one shot if functional weapon is equipped

  • Take a spaceship with a single weapon slot;
  • Equip weapon;
  • Shoot;
  • Check that spaceship’s round contains at least one shot;

unit-testing-at-the-speed-of-light-with-unity-test-tools-image12Test with test stub created manually.

FunctionalWeaponStub implements IWeapon interface but return hardcoded value.

unit-testing-at-the-speed-of-light-with-unity-test-tools-image01

Unlike Dummy object that does nothing, Stub contains hardcoded values and is intended to return them as this values result in specific execution path.

It is possible to create the same stub using NSubstitute:
unit-testing-at-the-speed-of-light-with-unity-test-tools-image02

The object created by Substitute.For implements IWeapon interface and is returned as IWeapon type. But it is actually a proxy object whose behaviour could be changed. Returns is an extension method of NSubstitute library that changes the behavior of this proxy. When Shoot method is called, the value specified in Returns() method is returned. It is also possible to provide a sequence of values or a delegate.

A sequence of values would be returned In the example below.
unit-testing-at-the-speed-of-light-with-unity-test-tools-image08

First call to randomNumberService.Range() with any parameters will result in 0. Next call will return 2 and so on.

Other useful function of NSubstitute is the ability to match arguments by template.

In the example below the default behavior would be overridden making any call to Range(10,100) return 80. For more details read the NSubstitute manual.

The approach with random number generator stub is highly effective in testing random events, because it is possible to emulate required sequence of random numbers.

Making stubs manually is easy and naming them correctly leads to creation of clean and readable tests. NSubstitute can decrease the amount of code and provide additional flexibility.

Test Spy

But what if it is needed to log the behavior of an object? How many hits has an opponent received? If the test double has logging functionality, then it is called a Test Spy.

Encounter means meeting someone in space en route to a star system. The player can choose an action to take in an encounter. For example, the police could be ignored and a pirate attacked.
Test scenario: make sure that opponent gets hit in an encounter

  • Take an opponent;
  • Take a spaceship with two weapons slots;
  • Equip two weapons;
  • Create an encounter;
  • Choose attack action in the encounter;
  • Check that opponent got hits;

unit-testing-at-the-speed-of-light-with-unity-test-tools-image05

Encounter requires two spaceships and a random number generator to calculate probable outcomes. AlwaysMaxRandomNumber is passed which will always return maximum number and this will leave no chances for the opponent to evade the hit. The test is following a very specific execution path.

The player spaceship is real in this encounter, but the opponent is a spy. The spy combines stub and logging functionality. It logs the hits that would be checked later in the test.

unit-testing-at-the-speed-of-light-with-unity-test-tools-image14

It is possible to assemble a test spy using NSubstitute’s Arg.Do<> and When… Do… constructs.

unit-testing-at-the-speed-of-light-with-unity-test-tools-image04

NSubstitute uses special argument matcher Arg.Do to execute a delegate when argument to it passes to the substitute. Which means that hitCount+=x.Count() would be executed on each method call.

Mock Object

The spy only logs the data, leaving verification to the programmer. What will happen if a spy is given the right to verify the data?

Test Spy with verification capability is called Mock object. (Note: This article shows non strict mocks.)

Test scenario: make sure that each weapon shoots when spaceship shoot method is called

  • Take a spaceship with two weapon slots;
  • Equip two weapons;
  • Shoot;
  • Check that each weapon got call to Shoot method.

unit-testing-at-the-speed-of-light-with-unity-test-tools-image11

This test case is special, not just because it uses Mock object pattern, but because it verifies the behaviour of the spaceship instead of the state. (Learn more about London and Chicago schools of TDD)

The test does not care about returning values of test doubles, it makes sure that certain methods with certain parameters were called. We use an assumption that if a spaceship has called correct methods with correct parameters on external systems, then it works correctly.

Making mocks manually might be rather time consuming. Using NSubstitute framework to get mock objects for cheap is a good choice.

NSubstitute can also verify if methods were called in a specific sequence.

unit-testing-at-the-speed-of-light-with-unity-test-tools-image03

A mock object not only logs the calls but also verifies them. It is a test spy with built-in assertions.

Fake object

But what if a test double needs some logic?

A test double that contains logic is called a Fake object and it is a dangerous beast. It is the only test double that contains logic and emulates a real system component. As the Fake itself is complex, it is used to substitute real system components which could not be substituted by stubs. The most common example is the substitution of a real database with an in-memory database or using fake web service instead of a real one.

The best solution is to avoid fake objects when possible, as they know too much about the behaviour of the component they substitute and contain logic that should be tested and is a subject to change as well.

Summary

As a brief conclusion:

  • Making stubs, dummies and spies manually is not complex and giving them proper names will make code readable and clean.
  • NSubstitutes is designed for mocking and using it is preferable to designing mock objects by hand.
  • Checking the behaviour and state based testing are both powerful. Use the one that makes the test simpler.
  • Avoid fakes (logic) in test doubles when possible.

Happy unit testing !

Examples are on the GitHub.

P.S.:

Different books use different terminology see terminology in books
NSubstitute is very powerful and taking a look at the documentation is a good idea.
Good principles to follow.
If you still don’t feel the difference between mocks and stubs read “Mocks aren’t Stubs”.
Arrange Act Assert (AAA) pattern was used to structure unit tests in the article.
There is an open source implementation of a Space Trader game worth playing with.

Comments (21)

Subscribe to comments
  1. Imi

    August 8, 2014 at 1:42 pm / 

    This post has nothing to do with Unity at all.. Heck, it even has nothing to do with games! (Beside the example theming, which is totally arbitrary and over-simplified).

    I am sure there are a couple of dozends intro-tutorial like this on most of the bigger unit-testing websites already. (Or was this post copied from somewhere?)

    What a wasted opportunity to speak about things like:
    - How to substitute unity systems, e.g. an mecanim Animator or a Physics collider
    - How to assert graphical things, which arrises very frequently in games
    - How to substitute the unity’s Input system
    - How to unit-test OnGUI logic code
    - How to not have the scene getting dirty all the time when unit tests create temporary GameObjects (used for stubbing, mocking etc.)

    Its a Unity-blog, guys! Talk about Unity!

  2. Thomas Petersen

    August 1, 2014 at 10:03 am / 

    @Ben: This is what you should use the integration testing part of the Unity Test Tools for. If you haven’t picked up the package, I suggest you do so and read Dmytro’s last blog post where he links to a great tutorial project he made called Growing Games Guided By Tests.

  3. Ricardo Amores

    July 31, 2014 at 9:59 am / 

    @BEN HUMPHREYS the unit test tools contains a set of MonoBehaviours to test game functionality (e.g. check if two objects actually collided)
    I’ve not tried those things yet though so the only advice I can give you is to download the tools and check them out.

  4. Ricardo Amores

    July 31, 2014 at 9:57 am / 

    @MSW it is more code, right but not that much more as you might think. And you do not duplicate code. Actually you can probably reduce it extracting common logic that you can share among several views ;)

    I’m actually writing a post partially related to this topic: using a MVC pattern for UI with unity3d, for my company blog, you might want to check it out (see the website and sorry for the shameless plug)

  5. Ben Humphreys

    July 31, 2014 at 9:27 am / 

    I checked out the unit tests and they seem useful for individual functions. I have one question though, how would I test anything larger than a class or two? Most errors I’ve encountered occur in scenes or between multiple scenes. I think tests on prefabs, scenes, instantiating and controlling GameObjects would be more useful. What’s the best approach for testing things in-situ?

  6. MSW

    July 31, 2014 at 1:36 am / 

    @RICARDO That makes sense, but it seems like a lot of extra code and duplication. I’m definitely interested in seeing some actual examples in the upcoming blog post though!

  7. Ricardo Amores

    July 30, 2014 at 4:59 pm / 

    @TOMAS thanks

    @MSW The approach we take is treating the MonoBehaviours as some kind of “View” (if you are familiar with the MVC pattern) Logic exists in .net plain classes, and that way you decouple the “logic” (how an object behaves) from the “view” (how it looks like in the game)

    That way you can test the code that contains the logic using the strategies discussed here, while you still are able to use the Unity3D editor (to some extends)

  8. Lior Tal

    July 30, 2014 at 4:22 pm / 

    @Dmitry – Thanks, looking forward to that.

    You don’t have to convince me, i know it’s possible.

    I’d love to have (or even help making) a “bible” or list of guidelines / patterns for doing it right and to get testable code.

    Most of the Unity code i’ve seen does not allow it and was not written with that in mind at all.

  9. Dmitriy Mindra

    July 30, 2014 at 2:35 pm / 

    @LIOR TAL Regarding frameworks that can substitute static and sealed classes, there are proprietary libraries that can do this magic using profiler API for .NET but not for Mono.

    Let me show you in my next blog post how we can build testable object oriented design for a game (with MonoBehaviours and GameObjects) and to convince you that such tools are not needed for proper unit testing.

  10. Dmitriy Mindra

    July 30, 2014 at 2:28 pm / 

    @LIOR TAL Yes, we were also considering other libraries. We chose NSubstitute because is an easy to use open source library that is widely used and has active community.

  11. Thomas Petersen

    July 30, 2014 at 11:58 am / 

    @Ricardo: I think we will always have the package on Asset Store for convenience, but we are also working on making it available in a different way. Stay tuned.

  12. Lior Tal

    July 30, 2014 at 11:37 am / 

    @Dmitriy – why did you choose NSubstitute to be bundled with UTT? did you consider any other framework?

    Are there other frameworks that do not have these limitation for mocking? (e.g: can mock any class, including sealed ones)

  13. Dmitriy Mindra

    July 30, 2014 at 11:18 am / 

    @MSW I’ll answer your question in my next blog post in 2-3 weeks.
    @ADAM No, we can substitute only abstractions and classes that could be inherited from. Both Rigidbody and BoxClollider are sealed classes and we can’t inherit from them.

  14. Ricardo Amores

    July 30, 2014 at 9:57 am / 

    @THOMAS when you say that I should keep an eye on this pages, you mean that you are thinking in a better way to share code libraries than the Asset Store?

  15. Adam Scura

    July 30, 2014 at 12:30 am / 

    Can we substitute Unity components like BoxCollider and Rigidbody?

  16. msw

    July 29, 2014 at 9:37 pm / 

    I’d like to see more about how you can properly use interfaces within Unity. The problem is that Unity is designed around MonoBehaviours, and if you try to use interfaces instead of MonoBehaviours you lose almost all the functionality that makes Unity useful: you can no longer drag and drop, see the inspector, have designers easily fill in values, make prefabs, serialize state automatically, use GetComponent, etc.

    Let’s say you’ve currently got a Spaceship MonoBehaviour that has two Weapon slots, and Weapon is a MonoBehaviour. And you have a bunch of prefabs with filled out values in the inspector, and the team can drag and drop those weapon prefabs onto spaceship weapon slots. If you change the Spaceship so it instead has two IWeapon slots, and have Weapon implement IWeapon, you lose the Weapon slots in the inspector and can no longer construct spaceships by dragging prefab weapons. How do you deal with this?

  17. Lior Tal

    July 29, 2014 at 8:10 pm / 

    @Thomas i do blog occasionally (less than i’d like to) about Unity and coding. I am by no means an expert on testing but i would like to promote and raise awareness to this topic, especially now that coding in Unity pays my bills now :)

  18. Thomas Petersen

    July 29, 2014 at 11:40 am / 

    @LIOR: You are spot on. We will be making more articles with that focus. There is a subtle hint in this one through the use of interfaces, but we need dedicated blogs for just this. We do sometimes have guest bloggers here… ;-)

    @RICKYAH: I would very much suggest you keep an eye open for these pages in the near future.

  19. RickyAH

    July 29, 2014 at 11:20 am / 

    @LIOR TAL I totally agree. One of Unity’s flaws is that it is extremely easy to just drop code without any control until things work. Test projects doesn’t encourage a proper architecture either. :(

  20. RickyAH

    July 29, 2014 at 11:16 am / 

    That’s a really nice addition, but what would be the best way to be notified of changes in the unity test tools? Checking out the Asset Store seems really inefficient to me. How about to, in addition to keeping the Unity package in the Asset store, to upload the code to github so we can get the code and updates in an easier way?

  21. Lior Tal

    July 28, 2014 at 11:53 pm / 

    Nice article. I think the major issue is not the technical details of how to work with NSubstitute, but more of how to structure your code, specifically with Unity, to be able to use it effectively.

    Most of the code i’ve seen would not allow unit testing it efficiently (or at all), so the harder part (in my opinion) is how to structure game code better for testability.

Comments are closed.