Unity’s test automation team
Hello everybody, my name is Elvis Alistar and I have been working with Unity for more than 2 years. I am responsible for leading the team of Software Development Engineers in Test (SDET), which is part of Unity’s QA department. As the job title implies, we are a team of software developers that absolutely love testing. In this post, I would like to let you know why it is so critical for Unity to have a team of expert developers focusing on test automation.
Unity is a rapidly growing company with more than 450 employees from 50 different nations, working in 27 locations. The wide-spread, distributed nature of our company means that we have to be able to work with developers on all the different time-zones, from different cultures and with different backgrounds. We also have more than 2.5 million registered developers and often we interact with them through our forums, feedback website, and alpha and beta groups and by handling some of the bug reports we receive from them. Close collaboration with our development teams means that good communication skills become key.
Unity supports 12 major platforms and a few smaller ones. Users can also use three different scripting languages for writing their game code. Multiply these with the fact that we have runtime tests, graphics tests, integration tests, unit tests, UI tests, performance tests, etc. and the number of different combinations our test automation infrastructure has to support and cope with is phenomenal. Take our runtime tests for example. We now have more than 1.300 unique Runtime API test cases. More than 13.000 test cases are run in a single automated build verification suite. We work in more than 100 development branches at the same time, which means that there are around 500.000 test points executed on our build farm every day. SDETs have to write, maintain, optimize and improve these on a daily basis.
Just two years ago, we only had a few test frameworks, which were not very well documented, we didn’t have too many tests and nobody owned, maintained or improved the frameworks or the test code. The SDET team was formed with the mission of taking ownership of these frameworks and drive the effort of cleaning up, documenting and optimizing all our tests. We also made sure that all developers know how to use these frameworks and that they can easily create new tests using them.
Unity’s code base is growing very fast and we need to make sure the product performs rock solid with every new release we are shipping. That is why we place great emphasis on test automation here at Unity. Just to put things in perspective, we now have somewhere around 2.2 million lines of code, out of which about 400.000 are test code spread over more than 7.000 files. This ensures that we can keep the quality bar high, avoid introducing new regressions and make sure that known bugs that were resolved will never show up in our product again.
Now that is what I call a challenge!
We are a distributed team of 8 developers working on tools, frameworks and feature automation, with a big emphasis on the latter. Two of us are located in Copenhagen (Denmark), one in Silicon Valley (USA) and five in Odessa (Ukraine). Our job is to figure out ways to improve the overall workflow for everyone involved in the development of Unity, by improving the automation on the build farm, making regression suites to prevent bugs flowing in and also evangelizing the usage of the frameworks to both testers and developers.
Unity is a very complex product and to help work more efficiently we have split it into different areas: Scripting, Lightmapping, 2D, Animation, etc. Every SDET is responsible for writing, maintaining and reviewing tests in one or more of these areas. New SDETs will usually start with one area and work on it until they become proficient with using our processes, frameworks and strategies. After they gain enough knowledge and build enough confidence they take on more areas. All SDETs eventually work on improving our existing frameworks or adding new ones. We have 7 different kinds of automation frameworks at Unity and SDETs have to be able to work with all of them and help anyone else that has to work with them.
Our job requires a strong collaboration with the other teams inside QA and across teams in our R&D department. We have to communicate well with developers and Software Test Engineers (STE) to make sure we know who is working on any given feature, which parts of it have been developed and tested and what needs to be automated. We all participate in QA’s main test activities on any given release, even if they don’t involve test automation: Exploratory Testing, Full Test Pass and Release Acceptance Testing (see Manual Testing session on our Testing Unity 4.0 blog post).
Our automation work is rarely tied to a specific Unity release and that gives us a lot of freedom and flexibility to focus on what we want to improve. This also allows us to travel to our different offices to assist developers or to work with other testers on location. We attend the bigger testing conferences and we speak at conferences whenever we have the chance. We participate in our now famous HackWeeks where we can experiment with whatever ideas and projects we want, no matter how outlandish they are.
Skills and attitude
As SDETs we are required to have extensive knowledge about programming, object oriented design and test automation (unit testing, TDD, high level tests, etc.). We have to be able to spread the word about testing frameworks and practices, offer great feedback in code reviews and be able to work well together with developers. We value clean code and development best practices.
At Unity, we also need to be able to work efficiently with many tools we use internally like FogBugz, QMetry, TeamCity, RhodeCode, Sikuli, NUnit, Moq, Unittest++, Atomiq and others.
So, there you have it! Being an SDET at Unity is both awesome, very rewarding, and the diversity of tasks we take part in is simply mind-blowing. The challenge is on! Are you up for it?