It’s been a fair while since we wrote a post about testing Unity, so we‘d like to update you on what’s going on in QA. This is the first of two posts from QA. I’ll post the second one soon, which focuses on our tools, cycles and analytics.
I assume you know about Unity if you’re reading our blog. As a developer, you’re intimately familiar with most of the features, but there are a few aspects from a tester’s point of view you might not have thought about before.
Eyes Wide Open
Unity is one of the most testable products I have ever worked on. Practically everything in Unity is available for a tester. It consists of a ton of APIs, profilers, logs etc. which enables us to do anything we want from code, including verifying that any action we have taken actually gets us the expected result. On top of that, Unity on the core parts requires only a single installation on a single machine, so the infrastructure required to run automation is very small.
All those platforms
There is one caveat though. A core feature of Unity is that it can build your games to many platforms, which naturally requires us to test such functionality. This does complicate a test rig quite a lot and it puts requirements on how our frameworks are built, so we can get the most bang for our testing buck.
From a more general point of view, we have the problem of covering all those platforms both during a regular test run, but also when you have submitted a bug saying “this and this phone has given us this specific problem”. Or different browser versions on different OS’, or specific graphics cards etc etc. You get the drift.
Let’s not forget about the love we get from our community. In so many ways you guys are the backbone of Unity, adding true value through the forums, answers and testing cycles. This is something we take very seriously, something which commits us to be responsive and helpful.
No, not the large productions. Automation, automation, automation. We have the testable product, we have the infrastructure and we have the need. Were we to make a single cycle product, and then move on to the next, we would not be taking such a bet on automation, but features in Unity live for years and repeated manual testing on such a product simply doesn’t make sense.
Use the community
We have you guys. You’re a great help as it is, but we can do a whole lot more. We can help you with tools which will help you test your own games. Let’s face it; while we believe we’re doing a heroic effort to cover everything, it is practically impossible.
Tools might include better overviews of bug reports, automated duplicate finders when you submit bugs, integration to answers. Even something like publishing testing suites for you to run on your own games could be an option. Helping you will go a long way in helping us, if we do it in a way where results can be piped back to us in a way so we can take action on it.
Good internal reporting
This is about making reports for the Unity development team. We need to assess the quality of different parts of Unity, which parts have good coverage, a high concentration of bugs, or where users are experiencing a lot of pain.
This point is very easy to say, it’s also conceptually easy to lay out a plan for which type of reporting you want, but the implementation of such a metric based regime is very hard. It requires us to use the same terminology across all of our tools, capture our data consistently and on a regular basis and then build the reporting needed on top of them. We’re currently in the phase of implementing the same terminology across our testing tools right now.
Everything starts with the people. We are not stronger than the people we hire, the people we retain and the people who make it all happen. At the time of writing there are 17 engineers and student workers in the Unity QA team, but we are hiring many more.
Where we are
Currently, testing in Unity is based in Copenhagen, Vilnius and Brighton. We are also recruiting for a new office in Odessa.
In Copenhagen we have mostly been focusing on the editor and the core, since this is where most of those developers are placed. Vilnius is mostly about handhelds and consoles and Brighton has been taking care of the Webplayer. I say mostly, because at times everyone is spread across several features, for example, PS3 testing is now in Brighton.
As you can see, all of Unity is spread across the globe. One of the great things about the QA team is that we are constantly in communication with each other via a big Skype channel. 27 people participating, with regular eavesdropping by the devs and the support team.
Roles in test
We have some distinct roles in test, which helps us define our recruiting better, but being in test you often have to have a wider scope on what you do than when you are a developer.
Software Development Engineers in Test are developers working in test. These guys are working on tools, frameworks and feature automation all at once. Their job is often to figure out ways to improve the overall workflow for everyone involved in the development of Unity, by improving the automation on the build farm, making regression suites to prevent bugs flowing in and also evangelizing the usage of the frameworks for both testers and developers. They are the backbone of the AAA strategy.
Software Test Engineers are what you would call “manual testers”, but that is not a fair term. Beside figuring out how to actually get proper coverage on a feature, a task which requires a fair amount of technical knowledge, these guys also need to be good at creating an overview of an area . They are also the very first “users” of features, able to provide feedback very early in the lifecycle and ensure that we have as good a first shot as possible for our alpha users. They are the backbone of a good internal reporting strategy.
QA Student Workers are using their time on looking at the bugs you submit, try to reproduce each and every one of the and then forward the bug to the developers. These guys are doing a heroic effort to respond to your report. If they can’t figure out what is going on, they forward it to a QA within the area or send a reply to you asking for more information. It is a relatively new role in Unity, which we started in the spring 2012 and they are a big part of our effort to be responsive to our community.
We have the best community in the world. A bold statement, but I believe it to be true. I have never been in a company where the community has had such a big influence and impact. I’ve said it earlier in this blog post, but it is something we value deeply. We use the community for many things and I’ll highlight a few here.
Sometimes we assemble alpha groups. The alpha lists are comprised of a small number of people who have a track record of giving us good feedback on features. They are given drops earlier than anyone else and we ask them to take a build (often broken in some areas) and help us make sure we make the right features for the next versions. This feedback is what we use to change any issues we might have with how you guys use it.
The community also helps us very visibly with reporting bugs to us, both during beta tests and after released versions. Beta tests run for a considerable time, through several releases and we gather tons of bugs from you guys. The beta testers who are really good at finding AND reporting bugs are also given some swag for their efforts (finding them is good, but to make a great beta tester you should consistently deliver easy to reproduce bug reports. More on this in part two).
Finally I want to mention the willingness to help us when we ask for it in one of our beta groups. I recently called for very large projects from real-life production teams and got a response within one working day. There is no end to how much we appreciate that kind of response and participation.
I think our lead developer Lucas Meijer articulated all my thoughts about this challenge when he stated that “writing automated tests is very fucking hard”. It is a job which requires you to know a lot about testing techniques, infrastructure, the product, and be a coder who can write extremely solid code. A guy such as Lucas. It is interesting how this task is frowned upon when you look at the challenges it poses. I have met a lot of good developers in my career, but the stars of any dev team were also those that had a great aptitude for testing. I guess this little rant is to say that you should not expect “to start as a developer in test and then grow into a developer”. It’s the other way around.
It is also a difficult job to be a great software test engineer: you need to possess the correct mix of technical knowledge, and communication and organizational skills. It’s a rare mix, and we continue to look for the right candidates.
If you’re that guy or girl, if you want to be the one making everything run smoothly in a feature team and if you have a passion for quality you can’t stop preaching about, join us: http://www.unity3d.com/company/jobs .