Build Engineering and Infrastructure: How Unity Does It

October 21, 2011 in Technology

Hello!  I’m Na’Tosha and I’m the Build and Infrastructure Developer here at Unity Technologies.  While speaking with users at the awesome Unite 2011, I had several people ask me to write a blog post about how Unity Technologies manages to develop, build, and test Unity.

We do this with a combination of:

  • Continuous Integration
  • Automated Testing
  • Code Hosting
  • Code Reviews
  • Manual Testing

I’ll talk about how we do the first four of those things here.

Continuous Integration

The Continuous Integration Server

 

TeamCity

Our builds and automated tests are all run by TeamCity.

We use TeamCity from JetBrains as our continuous integration solution.  One of the things we like best about TeamCity is that it has a very clear UI that gives the end user a good indication of the state of a project.  For testsuites, it’s easy to see at a glance whether the suite passes or fails and what tests are failing.  It can farm builds out to agents on Windows, OS X, and Linux, and keeps an extensive history of builds and tests, which are very useful when trying to figure out why a test is failing or whether there is something wrong with an agent.  Generally speaking, we are quite happy with the feature-set and also with the support we’ve received from JetBrains.

Our only real complaint now is regarding the web UI — it can still be slow to load at times (specifically, when loading a large project page that hasn’t been loaded for a while, or when opening the custom build run dialog when a lot of agents are attached).  It can also be difficult regarding managing multiple branch projects, as there is currently no way to synchronize changes between branches that are copies of each other.  However, the projects are stored in XML, and the schema is not complicated, so I have written a tool to do this instead by modifying the XML directly.

The Build Farm

TeamCity Agents Page

TeamCity farms builds out to a distributed build farm of between 40 and 50 machines.

Along with TeamCity, we have a build farm of approximately 45 machines, known as build agents.  Most of these are virtual machines running Windows, Mac OS X, and Linux.  We also have a few physical machines used for graphics tests, because testing graphics functionality requires a level of GPU manipulation that we could not achieve with virtual machines.  Our virtual build agents are virtualized with VMWare Workstation on Linux hosts, using a combination of Apple and non-Apple hardware.

The reason for choosing virtual machines, despite the performance hit it gives us, is that virtualization allows us to easily maintain the system — instead of updating 45 machines separately, we update the template virtual machine image for a given operating system, then copy it to the live buildservers.  It also allows us to make every agent appear to be the same hardware-wise, even if they are running on different types of host hardware.  This is important because as we add more build agents, they will be running on different generations of hardware, and for the purposes of testing, we need them to all appear the same.

Automated Testing

TeamCity Tests

TeamCity provides a very clear interface for seeing the status of test suites.

We have several automated test suites that we run on a regular basis.  These are used, along with manual testing, as a metric for how stable our mainline codebase is, as well as when a feature branch is stable enough to be merged back into trunk.  We currently have four types of test suites:

  • Unit Tests — these test the correctness of individual functions / small bits of functionality in the code, but they do not test high-level features of the product.  These are used by developers, but not by our continuous integration server.
  • Graphics Tests — these work by building a Unity project that renders a set of static scenes.  The project is then run on the target platform (desktop, console, or mobile), and screenshots of each scene are saved.  We then compare the saved screenshots against a set of “known good” screenshots.  If the difference is greater than a certain amount, we assert that the test has “failed”.
  • Functional Tests — these work by launching one instance of the editor or player, and testing various aspects of its functionality.
  • Integration Tests — these are the most high level tests.  They work by launching an instance of the editor or player, testing a particular action or series of actions, and shutting down the editor or player and going on to the next test.

Our integration tests are executed by our continuous integration server as NUnit tests.  TeamCity can run MSBuild projects directly, and we have a script that runs them with Mono’s Xbuild on Mac OS X.  We developed our own framework for running the graphics tests.

Regression rig

Our regression rig tells us if anything has changed in the playback of previously-recorded content.

We also have a regression rig that we developed ourselves.  The regression rig, similar to the graphics tests, compares content played in Unity against previously recorded content, and checks for regressions.  This is a good way to catch high-level regressions (for example audio having become bit-shifted).

Our test suites are always being expanded, and, along with our regression rig, they have helped us catch a number of regressions and bugs.

Code Hosting and Reviews

Like many software companies, Unity Technologies has run through a few different version control systems.  In the beginning, there was no version control.  Eventually the first Unity Ninjas began using CVS, and eventually Subversion. About a year ago, we started to investigate distributed version control systems and eventually settled on Mercurial.  Now all of our source code is versioned in Mercurial, except for our public Mono, MonoDevelop, and Boo-related repositories on GitHub.

Why Mercurial?

Well, we had a few requirements.  The first was that the version control system work with our continuous integration server, which is TeamCity, so we started off considering Git, Mercurial, and Bazaar, and Perforce.  We also wanted something distributed because we have developers working in several different locations, and we also work on multiple platforms.  A distributed system allows our developers around the globe to interact with the remote server less frequently, and it also allows all of us to easily test our changes on multiple local machines without having to share changes that potentially break some other platform with each other.  We also wanted to be able to do feature development in branches and merge them back together successfully.  So we were left with Git, Mercurial, and Bazaar.  We spent some time evaluating these three systems.  We were interested in:

  1. A simple, easy to use and understand command-line interface
  2. Good GUI tools for the system on OS X and Windows
  3. A good code review tool that works well with the system.

We also wanted a system that we felt had a lot of momentum — an ecosystem that is growing and developing around it.  After a few weeks of testing, we eventually decided on Mercurial because:

  1. It was substantially more simple to learn and use than Git
  2. It had good GUI tools for both Windows (TortoiseHg) and Mac (SourceTree, and now TortoiseHg)
  3. It had a couple of different options for large-scale code review tools
  4. It had a good-sized user-base (which is growing), a regular development cycle, and seemed to be well-adopted by both open-source and and commercial projects

Another very big win for Mercurial is that it was the only DVCS (at the time, at least) that had even attempted to handle the issue of large binaries in the repository.  This was done through a few different publicly-available extensions to Mercurial.  Distributed version control systems, by nature, don’t lend themselves well to codebases with a lot of large binaries, so we knew we would need some system that can cooperate with the version control system to store the large binaries outside of the repository, but still let us version them in some way.  Not having to develop our own system completely from scratch was a really big win.

How to Host Mercurial and What to Use for Code Reviews?

After we decided on Mercurial, we needed to figure out how to host it and how we would review code.  With this switch, we also wanted to implement a new development policy.  Up until this point, we had always worked with just one central copy of the codebase, except for when we branched for release.  This led to something almost always being broken, which had a serious effect on productivity.  We wanted to be able to easily do feature development in branches; perform peer code reviews, build, and test verification on the branch; then merge it back into the mainline repository once it is complete.  The goal is that trunk is always in a releasable, or nearly releasable, state.

Code Review in Kiln

A Code Review in Kiln.

 

We looked at a few different code review systems, and eventually settled on Kiln from Fog Creek Software.  Many people already know that we have used FogBugz as our issue tracking system for a while now.  While FogBugz has some room for improvement (specifically, it doesn’t work very well as a public-facing bug tracker), it has done a pretty good job of serving our needs, and we have a really large amount of data in the system.  At this point, we would need a pretty compelling reason to put in the effort to move all of that data to a new system.  Kiln is a Mercurial code hosting system that interfaces with FogBugz and provides nice web-browsing of the repositories, code reviews on a per-changeset basis with one or more reviewers, and a server-side implementation of a Mercurial extension that handles large binaries. We have had some ups and downs with Kiln, mostly with regards to performance.  Our repository size, which is about 1.6 GB with a clean working copy, and has several fairly large binaries, as well as our number of concurrent developers, about 65 and growing, and our build farm, another 45 or so machines, seem to have pushed it to its limits performance-wise.  The self-hosted version of Kiln is currently not built with scalability to large teams (with large repositories) in mind, which results in very slow clones & pushes when there is heavy load. Hopefully, this will get resolved in the future. We’re not sure what our own future is with Kiln, but I will say that its feature set is quite nice, and it has allowed us to move to our desired development model of feature development in branches rather than in mainline.  Beware that it is written in .NET and does not run on Mono, so if you want to consider Kiln, you will need to run a Windows server.  I can say, however, that the Fog Creek support staff has spent countless hours trying to help us work through our various issues with Kiln.

Conclusion

Building and testing for so many platforms is a really difficult task — especially as our development team is rapidly growing, and we find ourselves putting strains on infrastructure and processes in areas we didn’t expect.  These tools and processes are fundamental to how the development team gets things done here at Unity.

Comments (17)

Subscribe to comments
  1. Na'Tosha Bard

    November 14, 2011 at 3:50 am / 

    @MandyCapy – please keep comments on-topic.

  2. Aras Pranckevičius

    November 12, 2011 at 4:24 pm / 

    @MandyCapy: what does this have to do with the topic of this blog post?

  3. MandyCapy

    November 12, 2011 at 3:44 pm / 

    Question

    Because Flash is so now at the end, it is high time the Unity, WebGL support. Let’s be honest. Flash is increasingly disappearing from the Internet. Unity must therefore put more on WebGL and offer an export possibility for WebGL.

    That would make if Unity, Unity will revolutionize the web.

    Please! makes it!

  4. Mike Currington

    October 30, 2011 at 6:46 pm / 

    Two versioning systems will bite you badly. Keeping two repositories in sync when you start branching builds for various purposes is at best a pain, and at worse a nightmare.
    At times you will want to be able to submit binary files at the same time as the source that uses/describes them – such as submitting shader changes at the same time as their matched models/textures. Running continuous integration tests and bug tracking integration are also a lot easier when change list numbers and descriptions are consistant across art and code.

    Perforce handles both binary and source/text files very well, if you are looking at commercial options.

  5. Sean Timarco Baggaley

    October 30, 2011 at 1:19 pm / 

    I have wondered in the past where it might be better to have two versioning systems instead of one. (No, wait! Hear me out, before you throw me out!)

    Most versioning systems, like git, Mercurial, etc., are usually geared heavily in favour of managing text assets, like source code. Stick with such systems for the actual source code-related stuff, but don’t use it for storing artwork or other media: these systems lack decent tools for artists and musicians, so you’re just making life harder for them. You need the right tools for the job…

    Instead, artists would use another system, like Alienbrain, which is geared much more heavily towards their medium. Thanks to the movie and TV industries, there’s no shortage of media asset management systems on the market, particularly for both audio and video assets. (iBase’s Trinity, for example.) So that takes care of that.

    When you run the scheduled build process, you can use scripts to pluck the art, audio and video assets from their respective management servers for packing into the installers.

    Granted, this is a more expensive approach, and it does make the Build Manager’s job a bit harder as you have more components to worry about, but it should (in theory) provide a much more streamlined—and productive—user experience for the developers and media creators.

    (And now, back to work…)

  6. Na'Tosha Bard

    October 27, 2011 at 10:53 am / 

    @Rod: We have been using an extension called “kbfiles”, which was a fork of the original “bfiles” extension for Mercurial. I’m happy to say, however, that this extension has been forked again, into the “Largefiles” extension for Mercurial and is now officially supported and included with Mercurial as of version 2.0 (which will be released on November 1st). Read more at: http://mercurial.selenic.com/wiki/LargefilesExtension

  7. Rod

    October 27, 2011 at 10:50 am / 

    So, you use a modified version of Mercurial? What is the plugin you use?

  8. Na'Tosha Bard

    October 25, 2011 at 6:53 am / 

    Guys, please keep comments on-topic. Thanks.

  9. Georges Paz

    October 24, 2011 at 3:24 am / 

    I rather suspect unity will add user login to the blog and ban this kind of annoying behavior.

  10. hippocoder

    October 23, 2011 at 12:28 pm / 

    I suspect eventually Unity will give up and give you those.

  11. Juan Sebastian

    October 22, 2011 at 12:30 am / 

    Nice post Natosha! :D
    Cheers!
    Juan the little guy from colombia :P ;)

  12. Robert Stehwien

    October 21, 2011 at 5:53 pm / 

    @Aras The focus of my complaint wasn’t art assets it was any sort of versioning support of metadata requires Pro and 3.5 was only planning on supporting SVN and Perforce in Unity itself. My dig was really “so you do all this internally but make it so hard for the users of your product to do it”. I always assumed that since Unity was so hard to put under version control, the developers were unfamiliar with it or were used to very small shops (obviously not the case as Unity has exploded with growth).

    Mercurial mentions above it has plugins so does GIT.
    http://stackoverflow.com/questions/540535/managing-large-binary-files-with-git
    http://caca.zoy.org/wiki/git-bigfiles
    http://git-annex.branchable.com/
    https://github.com/apenwarr/bup

  13. Alonzo

    October 21, 2011 at 5:02 pm / 

    What do recommend for simulating keystrokes and clicks in the integration tests?

  14. Aras Pranckevičius

    October 21, 2011 at 3:33 pm / 

    @Robert: you can use hg or git to version Unity projects just fine. The trouble is, however: what will git do when your artist has committed 20 versions of a 50 megabyte .PSD file? That’s like 1GB of storage for that one file on everyone’s disk. Now make that two hundred .PSD files. This is not a Unity-specific problem; git is just not meant to store large binary files, but they are everywhere among your game assets.

  15. Robert Stehwien

    October 21, 2011 at 2:16 pm / 

    Very good article on fundamental processes and things I’m used to doing in enterprise development. But it begs the questions: Why is it so hard to do those things with Unity projects? And why do you have to pay for pro to even get close?

    I’m hoping you add mecurial and/or git to the list of supported VCS in 3.5.

  16. Martijn Zandvliet

    October 21, 2011 at 12:50 pm / 

    Bookmarked for future reference! I’m pretty sure this will come in handy when I get to work with larger teams at some point in the future. :)

  17. Derek

    October 21, 2011 at 12:11 pm / 

    Brilliant article outlining some of the most fundamental processes in a large software project. Lots to examine here for our own processes. Good job and thanks!

Comments are closed.