Categories & Tags
Archive

Unity QA status for 4.2

May 9, 2013 in Company News and Info by

Our next upcoming release is Unity 4.2. Without actually going into what it’s going to contain in terms of features, let’s take a look at the other dimension we work along: Quality!

We just released 4.2 beta 2 this week and I will give you an honest status on the quality and bug situation in Unity.

Current situation

Our rapid iteration for 4.2 took a hit because a lot of our developers attended GDC 2013 and we had some issues with an update of our build software.

As of writing this, we have 40 bugs to fix before we will be able to call it a release candidate, but we will naturally get some bugs during the beta phase we need to fix, so a reasonable call is 60 bugs left. How long it takes to get them done is a different matter on which I shall not speculate here.

Burnup

It is good fun to take a look at how much we put into the release. From a QA perspective, this is one of the biggest bug fixing releases we have had.

We will hit 500 bugs fixed before 4.2 goes to final release. This includes 100 regressions from this or previous releases we have put in, so this release does bring  an improvement in quality as well as features.

42% of all the bugs fixed have been found and reported by a customer, which we are deeply grateful for.

4.2 Customer bugs

Since 4.2 has been in alpha for quite a while, we have already had a good number of incidents (bug reports users submit, which we convert to bugs if we can reproduce them).

196 have been reported, 14 are still not processed and 119 have been reproduced and converted to bugs. Out of those 119, 30 have been fixed, so bugs which are not of a high priority are getting postponed in order for us to have as little code churn as possible. It should be some comfort that we are releasing faster if your bug has not been fixed this time around.

Bug Clusters

From a QA perspective it is always very interesting to see where the bugs are clustering, because clusters unfortunately indicate more issues to be found. The report I made for this highlights overall bug clusters and clusters specifically from user submissions, so we can see if there are patterns of bugs where QA is not doing well enough.

In this graph I see no immediate warnings, which is a sign that we haven’t introduced any big and buggy areas; at least not areas where users have reported many bugs to us yet.

Looking at the entire codebase, we have the following picture:

The numbers show priorities 1-7, where 1 and 2 are the ones slated for fixing in 4.2. Assets management lights up, but it is also a very big area. In total we have just over  2000 active bugs, which is not scary in a codebase the size of Unity’s, but we need to be vigilant about keeping this number down.

So that’s the current situation with our codebase. With all the work being done to solve the highest priority bugs I hope you will find 4.2 to be yet another improvement in our effort to give you the best and most productive tool for producing your game.

Share this post

Comments (25)

Comments are closed.

EricJ
9 May 2013, 8:22 am

Awesome, thanks for the insight into the process. But of course now I want to know what’s under the blackouts.

9 May 2013, 8:30 am

It’s just great that you put such an honest status reports of your code base. It means you trust the public which is scary and hard and great of course and means you are confident in your team and product which is again great. Congratulation for such a great strategy.

Mike
9 May 2013, 8:48 am

You didn’t have to black out “PlayStation 4″ and “Xbox Fusion”. It’s okay. Everyone already knows.

9 May 2013, 11:40 am

Thanks!

Jason
9 May 2013, 12:58 pm

Will “Interactive Indirect illumination” be available for Unity 4.2 ?

9 May 2013, 5:54 pm

It’s be really awesome if you guys could let us know what planned features are coming in 4.2, this whole “ohh it’s a seecret” only really works for games. I’d specifically like to know about the shader editor and garbage collection optimizations.

ImaginaryHuman
9 May 2013, 7:08 pm

When will the 2000 bugs get fixed?

Nathan Warden
9 May 2013, 9:42 pm

Thanks for sharing! I try and read all of the QA blog posts :)

@ImaginaryHuman There will always be bugs in Unity, that’s just the nature of having a large codebase. I do know that they’re working hard on getting those bugs fixed as fast as possible as no sane developer wants to have any bugs in their software, so you can be sure they want to fix them :)

Kyle Hodgins
9 May 2013, 10:26 pm

I would like to know why you people have managed to illegally sneak your software into my Hard Drive without my Permission No wonder my Computer is slowing down it is Third party software like you sneaking into people’s Hard drives that really pushes my nerves.

Do it again and i will file a Report with the Authorities. Plain and Simple.

Thomas Petersen
10 May 2013, 12:34 am

For the guys requesting insight into the features, I will let others from Unity do that.

@ImaginaryHuman: We are shaving it down, but there will always be bugs. Sometimes a bug is just less important than working on a new feature, even for our customers.

@Kyle: We have absolutely no interest in having our software on your computer if you don’t want it, so it was not us. Please contact support@unity3d.com to get help figuring out what happened.

Klemens Forster
10 May 2013, 3:50 am

I wish support for OUYA!!

10 May 2013, 4:04 am

@Klemens Forster: there is already support for Ouya. It’s called Android and Ouya have a free pack with some helpful tools for Ouya.

10 May 2013, 4:06 am

Hey! Thanks, interesting, I appreciate it. Tomorrow I give a short talk at CodeMotion Berlin about game development with Unity. It suits. I’ll probably show it in order to explain some game dev issues.

10 May 2013, 5:21 am

One thing I noticed is that when I submit a report I have no idea if Unity QA tried to reproduce it and was not able to. I can handle being told my bug is not reproducible :) I just don’t like not knowing if it was ever looked at, this made me stop submitting bugs that require specific steps to reproduce.

Thomas Petersen
10 May 2013, 5:26 am

@Issam: We are currently not able to process everything we get, so we have a rating system on incoming bugs which tells us how much information is in the bug. Repro steps, project files, pictures and videos bump the rating of the bug report, so we process them from the highest rating because they tend to have a higher chance of being reproducible. Sorry if you have not received response on your bug reports.

Jason
10 May 2013, 9:19 am

Will “Interactive Indirect illumination” be available for Unity 4.2 ?.

Thomas Petersen
10 May 2013, 10:35 am

@Jason: “Without actually going into what it’s going to contain in terms of features,” surprisingly means I will actually not go into what it contains in terms of features. That is up to someone else to decide when the time is right.

10 May 2013, 1:40 pm

I appreciate there’s a lot of bugs and QA is working hard.

But don’t add any more features while there’s even 1 remaining bug. Finish all the bugs before any new features. All of them.

Thomas Petersen
10 May 2013, 4:10 pm

@Robert: While that is a noble intention, there are no companies in the world doing that. It simply doesn’t make sense for the users and thus not for the business. Even though I as a QA want the bar to be very high on quality, there is simply a tradeoff somewhere.

In this agile software world we are living in, there is a new thought on how to handle the bug process. The basic premise is to have a hard cap on the number of bugs, say 30, and then only put in a new bug in the bug tracker if said new bug is of higher priority than the current list. That way you would also only ever have 30 bugs to show to your customers.

Let’s just say I do not approve. I want all of it logged and tracked and be visible so informed decisions can be made. For this reason I want our bug situation to be visible to everyone, both internally and externally.

11 May 2013, 2:07 am

@Thomas i’m honored to have unity just because of your approach to QA, Awesome, REALLY AWESOME!
@Others, 2000 bugs does not mean 2000 killer stuff, one of them is that byte is being shown as checkbox in inspector (did you ever try to use public byte x in inspector?) another might be double click on title bar on windows 7 will crash unity if asset store window is open and is searching and a scene larger than 10MB is open (not a real bug just example) so don’t be scared by these numbers.
Some bugs are really not that important and probably there are others which yet are not tracked so are not in the list. 2000 is not scary for something like unity it’s really good. Take a look at bug fixes in huge softwares made by greatest engineers like softwares from Google and Microsoft and you’ll get a better idea, or pick up a testing/QA book to know more.

12 May 2013, 1:02 am

@Robert: “Finish all the bugs before any new features. All of them.” – a _lot_ of these 2000 bugs are really minor issues. E.g. typos in documentation; some button in editor UI getting cropped when your inspector width is smaller than X; etc. etc. Yes they are “bugs”, but I don’t think 50 programmers should stop doing any work whatsoever just because we still have a typo in the documentation.

That said, yes, we do have some areas that are “more problematic than others”, and we should improve them first. That I agree with. The “must have zero bugs, period” – I don’t agree with.

12 May 2013, 8:03 am

@ARAS As you pointed the problem seem to be the definition of bug in one’s head. I think as a growing community and considering what happend around releasing 4.0 and pricings and … and things like Mr. J***N and fluid surfaces/ilumination… Unity technologies and more knowledgable people in community should try to educate the population. Thanks for doing this.

Breakmachine
13 May 2013, 6:06 am

I have to agree with Chaoss Pierce, please start informing us earlier about which features we should expect in the next version. Been waiting for the new GUI-system for a looong time now. Shader editor aswell.

Tony
24 May 2013, 2:00 pm

Good, but what we really need is a less greedy license model from you guys.

Thomas Petersen
24 May 2013, 2:49 pm

@Tony: Most offtopic comment ever, but I’ll bite.

We have the exact same perpetual license available today as we did 3 days ago. Nothing changed there. Now there are just new options for those who need those. And fun fact, we also have perpetual license holders complaining that the new subscription is so much better value than the perpetual license they now hold.

So, we have more options for everyone. That’s it. No evil thoughts or hidden agendas. Just more choices. By the way, are you enjoying the free mobile add-ons?

Leave a Reply

Comments are closed.