Search Unity

Here’s a look at a new approach we’re trying for doing unit testing in a huge C++ legacy codebase that is largely hostile to testing.

We think a lot about testing. Both because we want to feel more comfortable changing things, and because we realize that many, many developers depend on Unity being stable and reliable. And that’s something we want to continuously get better at.

We have previously blogged about how we do high-level testing from C#, but over time we have come to realize that we also need much more low-level, much faster, and much more fine-grained tests. The thing is, though, that our C++ codebase is rather hostile to any such test. Testing a huge C++ codebase with lots of legacy code is tough.

But… we’ve now made it a little easier.

After looking around and realizing that no-one quite solves the problem the way we need to (what TypeMock does comes closest but it’s Windows-only so not of much use to us), we sat down and pondered what we could do. Obviously, we can’t rewrite our whole codebase for the sake of tests. Whatever we were to come up with would have to work with where we are and with how we write C++ code.

Hijacking…

This made us realize that the one thing we want is the ability to hijack any function anywhere in our tests. Take it over, mock it, fake it, stub it, rule it. We’re okay with function X calling function Y directly, as long as tests can hijack the thing and turn function Y into whatever they want.

So we thought “code patching!” Turns out we had just developed this nifty framework to do live code patching for the sake of profiling. But the problem is many platforms don’t allow runtime code patching, so that rules out that option.

… At Compile-Time …

So then we thought “no problem, we patch at compile time!” Tricky! Some guys have done it, like the Bullseye guys whose tech we use for C++ code coverage monitoring, but it’s still really quite involved. And we have many different compilers across a variety of platforms so… no. And our builds are already quite complicated.  So that one’s out as well.

… With Macros

With C++, if you have nothing else, you always have macros and templates. So I sat down, got my < and > keys ready and wrote us some templates sprinkled with some macro bits.

The proposition is actually pretty simple: once you have a mechanism to hook into any function, you can build a whole framework on top that reprograms those functions for whatever is needed in tests.

So first step: hooks. Goes like this:

Screen Shot 2015-11-24 at 11.02.19

It’s simple. Does nothing by default (and actually compiles to nothing in builds we ship) but when tests grab the hook, it turns into magic. There’s a bit of intrusiveness but it’s very modest.

Next step is to make the hook go live during a test:

Screen Shot 2015-11-24 at 11.02.37

This will look up the hook based on the function signature and then grab the hook for as long as the fake is in scope. 

So finally, we can program the hook:

Screen Shot 2015-11-24 at 11.02.51

From here, you can do all the usual mocking like redirecting the call to your own function, observing arguments and return values, and running the original function in a controlled way. And it’s straightforward to support instance methods in addition to free functions. In fact, the system makes it easy to even replace entire classes.

Screen Shot 2015-11-24 at 11.03.03

Conclusion

We have yet to see the full impact of this new approach over time, but we’ve already found that there are many tests that we can now write that we couldn’t write before. While we will not let up on functional testing, we hope that with a high-speed, comprehensive, and granular native test suite we further add to the robustness of the Unity core.

10 Comments

Subscribe to comments

Comments are closed.

  1. I guess finding useful, reliable inraomftion on the internet isn’t hopeless after all.

  2. “once we switch away from VS2010”, See, it’s not only us using older toolsets “because we own the pro/team version” (We relented and tried out 2015, it’s pretty good…)

    Thanks for the header shufty and the Mock primer, good to see that internal tools are moving forward.
    Heartwarming.

    1. “See, it’s not only us using older toolsets “because we own the pro/team version” (We relented and tried out 2015, it’s pretty good…)”

      Actually, we’d love to do the same and switch and the licenses are in place for that. The problem is partially the breadth of platforms involved in any such move. Especially consoles usually come with a specific compiler dictated by the platform. Often we can only make a partial move forward (like for the editor).

      Still, the happy day of moving past VS2010 shouldn’t be too far out :)

      1. I apeacpirte you taking to time to contribute That’s very helpful.

  3. Is it possible for you to post the C++ code that makes this work?
    Would be really interesting to see.

    1. We were thinking about open-sourcing the thing, but not sure whether we will find the time. Here’s the header for the central framework which should make it possible to see how the core mechanisms work.

      1. Awesome. Thank you.
        Would love to see it open sourced of course :)

      2. Artclies like this are an example of quick, helpful answers.

    2. You’ve captured this peyrfctle. Thanks for taking the time!

  4. Always nice to read more posts about testing. I think that for us as C# developers (with no access to the native engine internals) it’s important to see more updates that allow us to easily test our code.

    For example – faking MonoBehaviour derived types is impossible using standard open-source tools (like FakeItEasy and Moq) since they all rely on DynamicProxy and are limited to mocking certain types / methods.

    I’ve recently started looking at TypeMock but that is Windows only and i am not even sure it’ll work with Unity.

    I’d love to see more updates in that direction.