Categories & Tags
Archive

Testing Unity

May 8, 2012 in Technology by

It’s been a fair while since we wrote a post about testing Unity, so we‘d like to update you on what’s going on in QA. This is the first of two posts from QA. I’ll post the second one soon, which focuses on our tools, cycles and analytics.

The Product

I assume you know about Unity if you’re reading our blog. As a developer, you’re intimately familiar with most of the features, but there are a few aspects from a tester’s point of view you might not have thought about before.

Eyes Wide Open

Unity is one of the most testable products I have ever worked on. Practically everything in Unity is available for a tester. It consists of a ton of APIs, profilers, logs etc. which enables us to do anything we want from code, including verifying that any action we have taken actually gets us the expected result. On top of that, Unity on the core parts requires only a single installation on a single machine, so the infrastructure required to run automation is very small.

All those platforms

There is one caveat though. A core feature of Unity is that it can build your games to many platforms, which naturally requires us to test such functionality. This does complicate a test rig quite a lot and it puts requirements on how our frameworks are built, so we can get the most bang for our testing buck.

From a more general point of view, we have the problem of covering all those platforms both during a regular test run, but also when you have submitted a bug saying “this and this phone has given us this specific problem”. Or different browser versions on different OS’, or specific graphics cards etc etc. You get the drift.

The love

Let’s not forget about the love we get from our community. In so many ways you guys are the backbone of Unity, adding true value through the forums, answers and testing cycles. This is something we take very seriously, something which commits us to be responsive and helpful.

Our Strategy

AAA

No, not the large productions. Automation, automation, automation. We have the testable product, we have the infrastructure and we have the need. Were we to make a single cycle product, and then move on to the next, we would not be taking such a bet on automation, but features in Unity live for years and repeated manual testing on such a product simply doesn’t make sense.

Use the community

We have you guys. You’re a great help as it is, but we can do a whole lot more. We can help you with tools which will help you test your own games. Let’s face it; while we believe we’re doing a heroic effort to cover everything, it is practically impossible.

Tools might include better overviews of bug reports, automated duplicate finders when you submit bugs, integration to answers. Even something like publishing testing suites for you to run on your own games could be an option. Helping you will go a long way in helping us, if we do it in a way where results can be piped back to us in a way so we can take action on it.

Good internal reporting

This is about making reports for the Unity development team. We need to assess the quality of different parts of Unity, which parts have good coverage, a high concentration of bugs, or where users are experiencing a lot of pain.

This point is very easy to say, it’s also conceptually easy to lay out a plan for which type of reporting you want, but the implementation of such a metric based regime is very hard. It requires us to use the same terminology across all of our tools, capture our data consistently and on a regular basis and then build the reporting needed on top of them. We’re currently in the phase of implementing the same terminology across our testing tools right now.

The People

Everything starts with the people. We are not stronger than the people we hire, the people we retain and the people who make it all happen. At the time of writing there are 17 engineers and student workers in the Unity QA team, but we are hiring many more.

Where we are

Currently, testing in Unity is based in Copenhagen, Vilnius and Brighton. We are also recruiting for a new office in Odessa.

In Copenhagen we have mostly been focusing on the editor and the core, since this is where most of those developers are placed. Vilnius is mostly about handhelds and consoles and Brighton has been taking care of the Webplayer. I say mostly, because at times everyone is spread across several features, for example, PS3 testing is now in Brighton.

As you can see, all of Unity is spread across the globe. One of the great things about the QA team is that we are constantly in communication with each other via a big Skype channel. 27 people participating, with regular eavesdropping by the devs and the support team.

Roles in test

We have some distinct roles in test, which helps us define our recruiting better, but being in test you often have to have a wider scope on what you do than when you are a developer.

Software Development Engineers in Test are developers working in test. These guys are working on tools, frameworks and feature automation all at once. Their job is often to figure out ways to improve the overall workflow for everyone involved in the development of Unity, by improving the automation on the build farm, making regression suites to prevent bugs flowing in and also evangelizing the usage of the frameworks for both testers and developers. They are the backbone of the AAA strategy.

Software Test Engineers are what you would call “manual testers”, but that is not a fair term. Beside figuring out how to actually get proper coverage on a feature, a task which requires a fair amount of technical knowledge, these guys also need to be good at creating an overview of an area . They are also the very first “users” of features, able to provide feedback very early in the lifecycle and ensure that we have as good a first shot as possible for our alpha users. They are the backbone of a good internal reporting strategy.

QA Student Workers are using their time on looking at the bugs you submit, try to reproduce each and every one of the and then forward the bug to the developers. These guys are doing a heroic effort to respond to your report. If they can’t figure out what is going on, they forward it to a QA within the area or send a reply to you asking for more information. It is a relatively new role in Unity, which we started in the spring 2012 and they are a big part of our effort to be responsive to our community.

The community

We have the best community in the world. A bold statement, but I believe it to be true. I have never been in a company where the community has had such a big influence and impact. I’ve said it earlier in this blog post, but it is something we value deeply. We use the community for many things and I’ll highlight a few here.

Sometimes we assemble alpha groups. The alpha lists are comprised of a small number of people who have a track record of giving us good feedback on features. They are given drops earlier than anyone else and we ask them to take a build (often broken in some areas) and help us make sure we make the right features for the next versions. This feedback is what we use to change any issues we might have with how you guys use it.

The community also helps us very visibly with reporting bugs to us, both during beta tests and after released versions. Beta tests run for a considerable time, through several releases and we gather tons of bugs from you guys. The beta testers who are really good at finding AND reporting bugs are also given some swag for their efforts (finding them is good, but to make a great beta tester you should consistently deliver easy to reproduce bug reports. More on this in part two).

Finally I want to mention the willingness to help us when we ask for it in one of our beta groups. I recently called for very large projects from real-life production teams and got a response within one working day. There is no end to how much we appreciate that kind of response and participation.

Join us!

I think our lead developer Lucas Meijer articulated all my thoughts about this challenge when he stated that “writing automated tests is very fucking hard”. It is a job which requires you to know a lot about testing techniques, infrastructure, the product, and be a coder who can write extremely solid code. A guy such as Lucas. It is interesting how this task is frowned upon when you look at the challenges it poses. I have met a lot of good developers in my career, but the stars of any dev team were also those that had a great aptitude for testing. I guess this little rant is to say that you should not expect “to start as a developer in test and then grow into a developer”. It’s the other way around.

It is also a difficult job to be a great software test engineer: you need to possess the correct mix of technical knowledge, and communication and organizational skills. It’s a rare mix, and we continue to look for the right candidates.

If you’re that guy or girl, if you want to be the one making everything run smoothly in a feature team and if you have a passion for quality you can’t stop preaching about, join us: http://www.unity3d.com/company/jobs .

Share this post

Comments (13)

Comments are closed.

Alex
8 May 2012, 12:31 pm

“writing automated tests is very fucking hard”

This!

+1

Donavan
8 May 2012, 3:33 pm

As a developer that’s just made the transition to automated test developer. I agree 100% this is some of the hardest crap I’ve written.

Majestic
9 May 2012, 6:32 am

Yep! It’s very hard to find quality people, especially with such skills. Maybe alternative path is hire people with no or little experience and “educate” them in-house.

My opinion regarding Unity release cycle is that it should be occasional, like every month or similar, but divided in phases like feature preview, beta, release candidate with as needed iteration of each phase, so people would know what to expect (feature preview – no discussion about, no support whatsoever, just for preview; beta – limited “as is” support, bug submitting, fairly stable but not for production; release candidate – pretty stable, almost production support….)

10 May 2012, 7:45 pm

As an alpha/beta user, I can certify that Unity is on the right path!
They are building a very powerful solid groundbase for the future Unity.
To be better, bigger and stronger!

13 May 2012, 12:49 am

Are you guys ever going to fix monodevelop?

13 May 2012, 5:34 am

@Majestic: Doing education in-house is a valid path, but it requires a certain critical mass before it is viable. Regarding the release cycle, it is exactly the type of cycle we use now. The complexity of Unity just requires us to have much longer cycles than 1 month.

@AaronC: That is possibly the most off-topic comment I have ever seen. Please keep comments limited to the topic of the blog post.

Richard Fine
13 May 2012, 8:20 pm

On the topic of “tools to help users QA their own games in a way that feeds back to Unity,” what’s the story with Unity Player crash reporting these days? IIRC, at the moment when Unity crashes on Windows, it dumps a crash-report folder and pops up a “Please contact the developer” message box. That’s pretty a pretty unprofessional presentation (not that one can be *particularly* professional in the face of a crash, but still…) and I’m guessing that a lot of those crashes never get sent back to the developer.

You guys should do something similar to what Microsoft does with WER: run a crash log server. Developers sign up for an account and are given a crash ID for their project, which they plug into their project settings. When their game crashes in the field, Unity Player offers to automatically upload the crash log to the server (if the user clicks OK, which they often will – certainly more than emailing the devs). Developers can then sign into the crash log server and download their game’s crash logs to track down bugs. And, of course, one of the conditions of using the crash log server is that Unity Technologies *also* has access to the stored crash logs, so you guys can look for crashes that are coming from engine components.

14 May 2012, 2:30 am

Richard, that is a good example of a tool we can build to help game developers better. This particular one we have already discussed, but have not yet started work on.

15 May 2012, 2:00 am

Richard,
That’s indeed a very good idea (and way to improve Unity better).

Jason
21 May 2012, 3:36 am

Nice.
And don’t forget the interactive fluids surfaces and destructible geometry for the next unity 3D release.

Jason
29 May 2012, 5:40 am

It is also important to note that Unity must also start testing these things for the next release of Unity 3D:

– Destructible Geometry.
– Interactive Fluid Surfaces.
– Material Instancing.
– Visual Script Editor.
– Matinee style cinematic editor.
– Normal Mapping for terrain.
– Multi-texturing Shader.
– Realtime Reflections (so without a render texture).
– Voxel Clouds.
– Underwater god rays and caustics.
– Realtime Global illumination.
– More realistic glass shader.
– etc,etc,…

Thomas Petersen
29 May 2012, 6:29 am

@Jason: Maybe we need those features to actually be in the product before we test them. Keep the comments on topic, please.

28 Jul 2012, 2:37 am

Seems Interesting…

Leave a Reply

Comments are closed.