Categories & Tags
Archive

Moving the bug fight forward

October 29, 2010 in Community News and Info, Technology by

Testing might not be a part of the romantic picture we all have about game development. Often it’s more seen as a dirty job that someone has to do. At Unity we have a growing team of test engineers who really love their job and take pride in achieving excellence, and I’d like to share with you a little bit about how we work. I am guessing that most of you have some kind of interest in us doing our job as good as possible and that you’d like to know what happens when you open the bug reporter and click ‘Submit’.

BugReporter

Incoming
When you submit a bug report it goes into our Incoming Bug Queue. From there we do what we call sweeping. We weed out bug reports that contain nothing at all – no description, no information and no attached project, and we weed out the spam.

Investigation
All other bugs go into our Investigation Queue. In there we look at each case and simplify it until we get the cleanest scenario possible where the bug happens. If the steps are too unclear for us to reproduce the bug, or if we need a test project, we reply back to you asking for what we need. It’s also often a matter of the user doing something wrong or needing some help. In that case we respond with the assistance needed and add an entry to the documentation or answers.unity3d.com if needed.

Test Creation
The cases from investigation that are actually real bugs are then moved to our Test Creation Queue. Now what does that mean – didn’t we already test it? Well, it means that we take the simplified reproduction case and implement it in one of our automated test frameworks, depending on where the case fits. We have a few of those including our awesome Regression Rig. When the test is written, it will continuously run on multiple platforms. Whenever new code and functionality is committed, it will be held up against the growing list of tests. In theory this ensures that once a bug has been fixed and added to the automated tests, it will never show it’s ugly face again.

AutomatedTest

Fixing and verifying
From the test creation queue we end up with either writing a test or in some cases where this is not feasible, we make sure that the steps to reproduce the bugs are crystal clear. Then we assign the case to development. One of our highly specialized developers will take the case, fix it and send it back to us for verification. At this point the test written should always pass and the test department also verifies the fix manually. If something is not right, it’s back to the developer until we are sure the bug is really fixed.

It has taken us a long time to hone these processes – Unity has grown immensely since the early days when there wasn’t even a Test department but just a handful of programmers making the product, testing it and providing support. Unity is now vastly bigger in terms of features, users and people working on the product. This also means, that there was a time when our capacity for handling bugs and support cases was lower and we didn’t have the same processes. The test team is bigger now and armed to the teeth with sharpened minds, rockstar programming skills and a work horse mentality.

Onwards!
One of the ways we’re moving along now is drawing a line somewhere in the backlog of cases from when our capacity was lower. We have a bunch of old cases where the majority are duplicates of already fixed cases, cases with no information or cases that are obsolete because of changes in Unity. We will be focusing on the newer bugs, but there will also be something in that old pile that still has relevance. Chances are that these issues will have been reported again with Unity 2.6 or 3.0, but if you are sitting with an old bug report you submitted, that you believe is still relevant then you can help us get to it by submitting it again.

We are aiming for a workflow where we process each and every relevant case that gets opened and provide feedback as soon as possible. I hope you’re up for the task of helping us by submitting high quality bug reports and that you will continue to have a little patience and love for the Test Team – cause we love you!

Share this post

Comments (21)

Comments are closed.

29 Oct 2010, 2:49 am

Hello there, I think that a “reward policy” could push worldwide Unity users to do a better job in the bug report task.

I mean something like, if the thing you reported atually help us to improve Unity, you’ll be rewarded with…. something (it’s up to Unity Tech to decide what). It could be a discount for the next version, a free entrance to the Unity Conference an iPhone module or stuff like that.

Hope it helps

ciao

pixnlove
29 Oct 2010, 3:23 pm

Well it is very good to have an insight of what happened after we click that
submit button.
Compare to other software where you report bug 100 of time and the problem still occur without getting any form of feedback.
I have reported two bugs in unity2.6 and all of them were handled with great care and great support, and both of them were fixed.
I was not expected less than that from this superb team.
Thank you this post it really make users feel that they contribute to
the success of this wonderful piece of software.

Long life to Unity3 and his wonderful crew.

Jason Amstrad
30 Oct 2010, 4:43 am

I got an e-mail from the unity newsletter,and there was something about unity 3.1,which was going to be “big” (as they said the newsletter).

What do you mean by “big” in Unity 3D 3.1 ?

Nathali Abbortini
30 Oct 2010, 4:54 am

yeah,I got the same email.
What will be so big about Unity 3.1 ?
Can you give us a hint ?

30 Oct 2010, 1:11 pm

It will be a surprise! :)

Nathali Abbortini
30 Oct 2010, 6:02 pm

Ok ok,just a little hint.

Does it have to do with user interfaces ???? or ,.,..,???

Manon Seppen
30 Oct 2010, 6:06 pm

Maybe they are planning some high tech AI solution.
I hope they will integrate Autodesk Kynapse AI.
It would be a great addition to the Unity 3D package.

Herald Brunick
30 Oct 2010, 6:10 pm

I do not hope too much for things like ScaleForm GFX integration.
But I would very much like to see some very good AI solution in Unity3D ,and yes,Autodesk Kynapse AI would be a great addition to Unity 3D.
“Dynamic obstacle avoidance” , “NavMesh AI” ,etc,etc,…

31 Oct 2010, 2:44 am

@ Jason Amstrad, Elephants are big and sometimes they are in the room without anyone noticing :P

Manon Seppen
1 Nov 2010, 9:42 am

Then do not talk about elephants,talk about unity.

Nathali Abbortini
1 Nov 2010, 9:44 am

Agree.
Give us some hints about Unity,and not elephants.

1 Nov 2010, 11:31 am

It would be great to have visibility to where our bugs are in the process (Incoming Bug Queue, Investigation, etc.). Is any of that info visible to users today?

Cocoapod
1 Nov 2010, 12:34 pm

@Nathali Abbortini

We’ll soon find out maybe on Nov 10th Unite ’10 conference.

3 Nov 2010, 4:30 am

Some questions on bug reporting:

- Why is my project automatically attached for documentation bugs?
- Why do bug reporters never get feedback on their bugs they reported?
- Why are there not more fields to specify and categorize the bug?
- Where can i ask about the status of my reported bug?
- How are duplicates handled and reported back to the reporter?
- Where can i check the actual bug list to avoid posting duplicates?
- Where can i see my status on the Investigation Queue?

n0mad
4 Nov 2010, 10:59 am

Impressive process… Congratulations guys.
Also, all hail Test Engineers, they’re the ones who are responsible of perfection, litterally.
They’re also the first input in the features enhancement suggestion process.
Keep it on ramping up your workflow, Unity Team !

Manon Seppen
4 Nov 2010, 11:32 am

Everyone is talking about the new features in Unity.
How about some good solid “destructible geometry” technology in Unity 3D ?
Has anyone thought about that already at Unity technologies ?

Jason Amstrad
4 Nov 2010, 11:36 am

@Manon Seppen .

Nice thinking.
Nvidia APEX technology would be a great solution for destruction in Unity 3D.
By the way,congratulations on the new Allegorithmic Substance technology integration in Unity.

Brian Heemskerk
4 Nov 2010, 11:53 am

@Jason Amstrad , @Manon Seppen.

hello,i am a senior test-engineer at Nordic games and we have tested both Nvidia Apex and HAVOK destruction.
the HAVOK destruction technology is far more superior than Nvidia Apex.
the algorithm that HAVOK uses is much faster than Nvidia’s.
so i recommend HAVOK.

mike
16 Nov 2010, 1:18 pm

Nice article. For those who may be unfamiliar iwth the testing process, heres a quick overview: http://www.elementool.com/ebook/IssueTrackingeBook.pdf

L
24 Nov 2010, 3:50 pm

I would favor a Mono Develop that works realiably, a usable GUI and ironing out of the inconsistencies – any time – over adding new features. Especially when these features are just eye candy.

25 Dec 2010, 3:33 pm

It would be great to have visibility to where our bugs are in the process (Incoming Bug Queue, Investigation, etc.). Is any of that info visible to users today?

Leave a Reply

Comments are closed.