Categories & Tags
Archive

Testing Unity 4.0

November 14, 2012 in Technology by

In the aftermath of all the features and goodies Unity 4.0 will give you, I’d like to bring you the story of how the quality assurance, or “testing” if you prefer that, has commenced and evolved since the release of Unity 3.5.

Back in February when we parted with Unity 3.5, the QA department consisted of 11 souls, although that is including me and as a manager I excel at doing nothing. We had been through a battle for getting 3.5 out, mostly going through the bug reports our beloved community had sent us. In other words, we were primarily reactive and while we take pride in our community, we are not proud to let them do the majority of our testing and as the user base increases steadily, being reactive was not going to scale for long.

Manual Testing

In March we used a Ninja camp for trying out different test case management systems, which was a week well spent. We had trials of the systems and we had our hearts set on QMetry for a lightweight system which fulfilled our needs for reporting and tracking execution.

With the TCM in place we had a month of manual test case push to fill up our system with the test cases we needed in all the areas of Unity. Or at least some of them, because we were still short staffed for having coverage on everything. Soon after followed Full Test Pass 1 (executing all manual test cases spread on all platforms) on Unity 4.0, where we spread 480 test cases over many of our platforms, making the total number of test points (combination of a test case and a platform) in this pass somewhere below 3000. We had some areas covered too much and others not covered at all, but that is the nature of work in progress and it was completely open eyed we did that.

During the summer we got better acquainted with the new features introduced in 4.0 and more work went into our manual test cases. Some test cases were automated, some were discovered to actually have been automated all along and naturally more were added along the way. At this point in time we also had a good staff of student workers to handle incoming incidents from the community, so the STEs (Software Test Engineers, see my previous post on Testing Unity for further explanation) could be better focused on structured testing. As a result we had a little over 600 test cases to be executed in Full Test Pass 2 in August, which unfortunately landed 1 week before Unite for different reasons. So we actually only finished half of our designated 2800 test points. We did still uncover about 100 bugs in 5 days, so it was still beneficial.

Going forward to the release we had scheduled a Full Test Pass 3 immediately following RC1. Our initial smoke tests on RC1 revealed that we would not be able to get good data from this build, so we postponed FTP3 one week. Up till FTP3 our STEs did a great job of preparing all test suites, assigning them and spreading the load on all QAs, so we were ready to go from first thing Monday morning in the FTP3 week. 800 unique test cases, spread on 2800 test points and with a magnificent effort we got them all, except 18, executed in 5 days. We uncovered 90 bugs, many of which have now been fixed.

A Full Test Pass is one of the structured approaches we use to test Unity. But it is just one of the many tools we have in our toolbox. The bugs we found in these test passes are very often bugs we would never find using other techniques or just by monitoring the incoming bugs. It is an essential part of unearthing the problems in the code for us and it is great for regression testing.

During all this time in 4.0 we also had several full weeks with a pure focus on exploratory testing by everyone in QA. If you think exploratory testing is a matter of sitting down and behaving randomly, you would be massively mistaken. It is lightweight in preparation, but there HAS to be preparation. You need to select the area to focus on, which type of testing you want to execute on the area and then focus for 1-2 hours on finding as many bugs. It finds many bugs you would never encounter in a structured test, so it compliments it perfectly and in cases where you have little documentation or the feature is unstable, this is a very good technique to use.

Automation

During the year we have made leaps in automation. We had a pretty good starting point from what the developers had done, but the frameworks needed some more love and we wanted to move some of the test cases from an old framework to a new, faster and more lightweight framework. This new runtime framework also has the benefit of being executed on all of our platforms, so one written test case will execute the code in all of the targets. This is tremendously important when you realize that our platforms are effectively different codebases through a series of defines.

The work on the frameworks started slowly before we shipped 3.5, but with only 2 guys in our toolsmiths team, it got off slowly. In May we started ramping up a new office in Odessa with the primary purpose of increasing our test team in size and talent. The Ukrainian market for testers is quite good and I have had previous good experiences by recruiting both STEs and SDETs in Ukraine. During the summer and fall we managed to hire 6 guys in the Odessa team, of which one is working with the Toolsmiths. This has had a tremendous impact on our ability to produce automation and improve the frameworks.

To really ramp up on our automation, we had an automation push just after Unite. Almost everyone in the QA team got a good chance to push in automated test cases with the help of our toolsmiths, SDETs started doing code reviews on each other and we saw a steady increase in the number of test cases. What is also very interesting about automation is that this type of testing uncovers a different kind of bugs during the development than the two types of manual testing mentioned above.. Essentially you need to be extremely specific in your input and the expected results, so it is all about details. This is an added benefit of doing automation that few are aware of, but we saw it during this push. Obviously the benefit of having regression testing running on our builds is the biggest part of doing automation, but it is worth knowing that there is a different kind of benefit from it as well.

The automation effort is ongoing and we will ramp up even further on it with the following releases, where I will post some more data about it.

Incoming Incidents

The change in focus to being proactive instead of reactive might have meant that we slipped our handling of incoming incidents reported by our users. Truth be told, we have not handled the majority of our 3.5 incidents after 4.0 went into beta. One of the major problems with the incidents we get reported from the publicly released versions is that the quality of the bug reports is quite low. Almost 9 out of 10 reports we process end up being something other than an actionable bug. Contrast this with having 60-80% of our closed beta bugs being actionable and you can see why we change focus. The other side of the equation is equally in favor of the latest version: There is a much higher chance of getting the 4.0 bug fixed! Testing is all about prioritizing the effort in a completely endless amount of possibilities and thus most of it changes to the current release.

With those words, I’m very proud to say that we have processed 97% of all incidents reported by public and beta testers during the 4.0 release. In other words, if you report a bug during a beta, it WILL be looked at! I can assure you that this sometimes surprises the guys who report the bugs, since we often ask them for more information and they respond by “Omg, I didn’t think anyone would ever care”.

Well, we do! We care a whole damn lot about it.

Unity 4.0

So we have now released 4.0. We are happy and proud. But we are also QA. We have found many bugs which were not deemed important enough for this release, which is a reality of any kind of software, but it is also a little sting in the heart of any QA. Nevertheless, we believe that this release has a higher quality than any previous version released and on top of this it has some magnificent new features.

During this release more than 1800 bugs have been fixed. 481 of these were bugs found in previous version, so plenty of love for existing features as well. 673 of the total 1860 bugs were found and reported by our users.

QA is right now moving focus towards the next version we are going to release, ramping up on the new features we need to test and adding much more automation to keep regressions from slipping into your hands. If you want to join our ranks in the effort, go to http://www.unity3d.com/jobs and see if you match one of our positions.

Share this post

Comments (23)

Comments are closed.

Dad
14 Nov 2012, 2:45 pm

Great post! Very interesting and thanks for pushing so hard on quality. Bad bugs can stop a team in their tracks (Unity is currently unusable for current project on my laptop). If the test automation tools you are using are open source and available to others it would be great to hear what you ended up using for cross platform automated testing.

theuserbl
14 Nov 2012, 4:30 pm

Nice to see, that with Unity 3D 4.0 Linux is supported as output format.
Now it is possible to create with Unity Linux games.
But why is the Unity IDE, Unity Browser-Plugin, etc not created for Linux?
It exists only for Windows and MacOSX.
Is a Linux-port planned?

Robert Kabwe
14 Nov 2012, 7:46 pm

Fascinating! It’s easy to forget there are actual people behind the company. You’ve done great work and I can identify with the process, that sometimes it takes a lot of work beforehand just to figure out how you’re going to DO the work.

14 Nov 2012, 8:08 pm

@Dad: what’s the problem with Unity on your laptop? Does it work on a different laptop?

@TheUserBL: it’s a lot of work and has uncertain commercial benefits. We are excited to support Linux game development now and if it’s a roaring success, more support will surely follow.

@Robert: Thanks for the kind words, it’s a lot of work… but also really interesting work :)

Nicolas M
15 Nov 2012, 12:52 am

HL This not directly related to QA, bu I would like top have some data/graph/charts about Unity Feedbacks. How many were actually translated into new features? How many dropped? Did some of them inspired you to make new features (or gave you hints on how to work around some issues), etc.

I think this could be rewarding for the community to know that our feedbacks are really taken into account.

Kind Regards

Thomas Petersen
15 Nov 2012, 4:14 am

@Nicolas M: We don’t have the data to track feature requests into fixes, so I can’t do that. But we made the Linux player!

Alonso
15 Nov 2012, 4:19 am

When a client to Unity 3D-native IDE on Linux?… currently only support Mac and Windows :(

guest
15 Nov 2012, 4:35 am

@David Helgason: English isn’t my native but that means “No, we’re not going to support our software on Linux any time soon”.
Valve right now is pushing Linux game development. Guess what developers would like to make more Linux versions ? If you’d have the software already that will support Linux you’d give people some tools to play around. You would MAKE the game designer teams that way. At start they would use free version to gain some experiance and make some basic money then they’d buy Pro version. Thats how it works in indie titles usually, indifferent to platform.
There are alot of people that HAD to use Windows just because Unity doesn’t work with their main OS. You gave an option for Mac users why not for Linux, or do you think that people that use Mac has more developers, really ? There is certainly not more profit to gain from Mac deveopers than from Linux especially since alot of people would just go away from Windows version once their main OS is supported.

wyy
15 Nov 2012, 5:23 am

hello, my name is wangyanyu

Thomas Petersen
15 Nov 2012, 5:39 am

@Guest: Unity was started as a Mac only editor and since then ported it to Windows, so things were happening in the different order from what you suggest. Currently Windows accounts for more than 80% of our user base.

15 Nov 2012, 6:46 am

Thanks a lot for sharing this great information on your testing evolution. It is great that you are always evaluating new tech to improve current processes. I am not in QA, but sit right next to one in our dev team, so I always like to learn more so I can see the issues they deal with.

This is a noobish question, but what are converted bugs? Is that like turning a caterpillar into butterfly? XD

imaginaryhuman
15 Nov 2012, 6:57 am

I think it’s fair to say that any piece of software is hard to make totally bug-free, even small stuff. I can labor all I want thinking that I have covered all the bases but then find something I had no idea or didn’t even conceive of cropping up in an unusual scenario. Not that I’m an expert programmer, but I try to be quite thorough in making sure everything `looks right`. But something of the colossal size of Unity must surely be a massive programming undertaking, likely millions of lines of code?, so no hard feelings here about possible bugs that are obviously being hunted down as we speak. Obviously for some people a certain bug will more significantly affect their dreams than others but `overall` most people have it plain sailing. Thanks QA team for your constant vigilance!

Kasper Andersen
15 Nov 2012, 12:06 pm

@Scott: When something is submitted using the Unity Bugreporter it ends up in our bug tracker as an incident. We have an excellent team of Test Engineers and Student Workers that go through incoming incidents, and work on reproducing the problem described in each incident. When they know how to reproduce the problem, the incident is converted into a bug – and sent to triage.

We often get multiple incidents that describe the same bug. Distinguishing between incidents and bugs allow us to track both.

15 Nov 2012, 1:00 pm

@Nicholas M: You can actually see the features that made it into the product from the Feedback site here: http://feedback.unity3d.com/unity/all-categories/1/top/active

While it’s not exact, it seems that over 4% of suggested features have been either implemented (~3%) or are being implemented (~1%). And since we can assume that there are some pretty bad ideas in there and some problems that we’ve solved differently from how they’re suggested, it’s probably a pretty good track record. And from a number of votes perspective we certainly also come out even much better than just 4% too :)

15 Nov 2012, 1:04 pm

@”Guest”: Unity 4.0 supports building Linux games, so I’m not sure what you are worried about, except maybe developers who are so disinterested in making games that they won’t install Windows on a partition, run a VM, or buy a Mac Mini to develop games on (or any cheap PC, Unity’s not demanding in hardware and you can probably find a decent second hand computer for almost nothing). Frankly, and at the risk of offending someone, I’m not all that sad for people who don’t have a driving passion for creating cool things, except perhaps feeling sad about their lack of passion.

16 Nov 2012, 12:18 am

@David Helgason: I’d like to thank you and all your team for the efforts you’re putting on this amazing software. We’re “early adopters” of the engine (we started in 2007 with one of the first versions) and our company is almost completely focused on Unity based products. We waited a long for this 4.0 release (because we were obviously testing it since the first released beta version) and we’re now ready to create new amazing features because of it. So guys, congratulations for your new baby :) and please… continue in this way!

Sid Burn
16 Nov 2012, 1:53 am

@David Helgason
I started developing with Unity just a week ago, as a hobby. But before developing with games i worked and developed under Linux since 2003. I can only speak for me. I have the passion, that’s why i am using Unity under Windows. But developing or using Linux for near 10 year and than switching to windows is just a horror. I think, a lot of people want that they can use Unity directly under Linux. Me too. It doesn’t mean that they don’t develop or don’t have the passion.

I mean, it’s the same as if Mac OS X Support for the editor will be cancelled, probably a lot of people will be upset that they can’t work anymore with her favorite OS and all applications that they are used to be. And they will complain. But that doesn’t mean that they have no passion or will stop developing. And i think that telling all the people that than starting to complain to buy Windows and if they complain they have no passion is a little bit offending.

I will not complain. I think Unity is great, and i really like that it gets Linux Support for deploying. But i still hope that someday Unity itself will be running under Linux and i am not forced to do it under Windows.

16 Nov 2012, 12:21 pm

@Sid: When I found out about Unity, I was using Windows – and at that time, Unity was only available for the Mac. So, I had to buy a Mac first to be able to develop using Unity. That’s quite a difference in terms of investment compared to just installing Windows, possibly on a virtual machine. And like myself, a lot of people got Macs back then just to be able to use Unity (actually, pretty much everyone I talked to) … and I don’t know of anyone who regretted that.

Now you can use Unity on Windows or Mac – so it’s really easy to find a system that works well for you if you are one of the about 98% using either of these. I’m still using Unity on Mac OS X because that’s really a nice OS. Just recently, I installed Ubuntu on a virtual machine to test the new Linux builds and … well … it looks very much like Mac OS X ;-)

That said: This blog posting is about testing Unity 4.0. Not about Unity for Linux. But think about it this way: Having a Linux version of the Unity editor means that everything related to the editor needs to be tested on yet another platform. Which consumes a lot of resources which in the end would have to be paid somehow … and I believe it’s very unlikely that there’s enough people buying Unity Pro + Add ons for Linux to really make that a reasonable decision from a business perspective.

There’s another issue: A lot of the tools you’d usually use with Unity aren’t even available for Linux. Like, most 3D modeling tools (except Blender), Photoshop (of course, you could use Gimp), Substance Designer, Visual Studio (I’m using that on the Mac running in a VMWare Fusion Windows 8 installation ;-) ). Especially with 3D modeling tools that could create problems because Unity uses those tools for importing. So to some users, a Unity version for Linux could look crippled because there’s really no way to get certain imports to work properly because the required software simply isn’t available for Linux. With Cache Server, this may not really be a problem, though (artists using Windows or Mac would do the changes and importing and then you’d simply get the imported stuff from the Cache server).

So … I very much agree with David: If you’re passionate about developing games and you want to use Unity – simply get a Mac or install Windows, whichever you prefer. And if you’re really passionate about Linux – do something to increase its market share (which seemingly is around 1.5% at the moment, which basically means “almost no one is using it”, see also: Operating systems (Global marketshare)).

There’s hope, though, according to: Linux is the world’s fastest growing desktop OS – up 64% in 9 months (so maybe some day, having the Unity editor for Linux might actually become reasonable from a business perspective, i.e. the investment might be offset by the added income through Pro licenses, at least when the Linux folks will buy Pro lots of licenses instead of spamming the forums with complaints that Unity “free” has no realtime shadows ;-) ).

16 Nov 2012, 1:17 pm

I’m Part of the alpha/beta test program and let me tell you, they really add requested features, they listen to community and fix whatever bug is on their path. I’ve never seen such great support from an engine company. I guess i don’t even have to mention their names. xD

koblavi
17 Nov 2012, 5:07 am

Great post. This is what makes our community different, We know exactly what’s happening with our Releases. Proprietary project with an open source attitude. I don’t have 1500 bucks, but I love Unity to the bone!! And apart from the the awesomeness of the product and the great community, this is probably the next best thing: the Openness.

Alex
20 Nov 2012, 2:29 am

And what’s happened to the QA efforts for Unity 3.X during all of this? Presumably they’ve been diverted to 4.0 in their entirety:

- An Android crash bug I raised a month ago has yet to warrant a response
- A critical Shuriken bug has been erroneously closed as fixed and is still present in 4.0

https://fogbugz.unity3d.com/default.asp?494573_0tsn6lg6k003mnpb

    Thomas Petersen
    20 Nov 2012, 2:50 am

    @Alex: Like i stated in the article, the majority of our efforts in QA go towards the next version, simply because that is where we have the highest chance of getting something fixed.

    As for the Android bug you have submitted, there is no project attached and so the chance of getting something done about it will be very small. Essentially we prioritize cases where a small project, screenshot or video is attahched, since that increases our chance of reproducing it considerably. By the way, in an upcoming patch on 3.5 there will be quite a few Android fixes included.

    The Shuriken bug was reported fixed back in May, QA confirmed the fix with your repro project, but if you still see it in 4.0, please submit a bug report on it. That would be considered a regression from that time.

8 Feb 2013, 7:14 am

infonya mantap nih gan..thanks

Leave a Reply

Comments are closed.