Search Unity

Our next upcoming release is Unity 4.2. Without actually going into what it’s going to contain in terms of features, let’s take a look at the other dimension we work along: Quality!

We just released 4.2 beta 2 this week and I will give you an honest status on the quality and bug situation in Unity.

Current situation

Our rapid iteration for 4.2 took a hit because a lot of our developers attended GDC 2013 and we had some issues with an update of our build software.

As of writing this, we have 40 bugs to fix before we will be able to call it a release candidate, but we will naturally get some bugs during the beta phase we need to fix, so a reasonable call is 60 bugs left. How long it takes to get them done is a different matter on which I shall not speculate here.

Burnup

It is good fun to take a look at how much we put into the release. From a QA perspective, this is one of the biggest bug fixing releases we have had.

We will hit 500 bugs fixed before 4.2 goes to final release. This includes 100 regressions from this or previous releases we have put in, so this release does bring  an improvement in quality as well as features.

42% of all the bugs fixed have been found and reported by a customer, which we are deeply grateful for.

4.2 Customer bugs

Since 4.2 has been in alpha for quite a while, we have already had a good number of incidents (bug reports users submit, which we convert to bugs if we can reproduce them).

196 have been reported, 14 are still not processed and 119 have been reproduced and converted to bugs. Out of those 119, 30 have been fixed, so bugs which are not of a high priority are getting postponed in order for us to have as little code churn as possible. It should be some comfort that we are releasing faster if your bug has not been fixed this time around.

Bug Clusters

From a QA perspective it is always very interesting to see where the bugs are clustering, because clusters unfortunately indicate more issues to be found. The report I made for this highlights overall bug clusters and clusters specifically from user submissions, so we can see if there are patterns of bugs where QA is not doing well enough.

In this graph I see no immediate warnings, which is a sign that we haven’t introduced any big and buggy areas; at least not areas where users have reported many bugs to us yet.

Looking at the entire codebase, we have the following picture:

The numbers show priorities 1-7, where 1 and 2 are the ones slated for fixing in 4.2. Assets management lights up, but it is also a very big area. In total we have just over  2000 active bugs, which is not scary in a codebase the size of Unity’s, but we need to be vigilant about keeping this number down.

So that’s the current situation with our codebase. With all the work being done to solve the highest priority bugs I hope you will find 4.2 to be yet another improvement in our effort to give you the best and most productive tool for producing your game.

25 replies on “Unity QA status for 4.2”

@Tony: Most offtopic comment ever, but I’ll bite.

We have the exact same perpetual license available today as we did 3 days ago. Nothing changed there. Now there are just new options for those who need those. And fun fact, we also have perpetual license holders complaining that the new subscription is so much better value than the perpetual license they now hold.

So, we have more options for everyone. That’s it. No evil thoughts or hidden agendas. Just more choices. By the way, are you enjoying the free mobile add-ons?

I have to agree with Chaoss Pierce, please start informing us earlier about which features we should expect in the next version. Been waiting for the new GUI-system for a looong time now. Shader editor aswell.

@ARAS As you pointed the problem seem to be the definition of bug in one’s head. I think as a growing community and considering what happend around releasing 4.0 and pricings and … and things like Mr. J***N and fluid surfaces/ilumination… Unity technologies and more knowledgable people in community should try to educate the population. Thanks for doing this.

@Robert: “Finish all the bugs before any new features. All of them.” – a _lot_ of these 2000 bugs are really minor issues. E.g. typos in documentation; some button in editor UI getting cropped when your inspector width is smaller than X; etc. etc. Yes they are “bugs”, but I don’t think 50 programmers should stop doing any work whatsoever just because we still have a typo in the documentation.

That said, yes, we do have some areas that are “more problematic than others”, and we should improve them first. That I agree with. The “must have zero bugs, period” – I don’t agree with.

@Thomas i’m honored to have unity just because of your approach to QA, Awesome, REALLY AWESOME!
@Others, 2000 bugs does not mean 2000 killer stuff, one of them is that byte is being shown as checkbox in inspector (did you ever try to use public byte x in inspector?) another might be double click on title bar on windows 7 will crash unity if asset store window is open and is searching and a scene larger than 10MB is open (not a real bug just example) so don’t be scared by these numbers.
Some bugs are really not that important and probably there are others which yet are not tracked so are not in the list. 2000 is not scary for something like unity it’s really good. Take a look at bug fixes in huge softwares made by greatest engineers like softwares from Google and Microsoft and you’ll get a better idea, or pick up a testing/QA book to know more.

I appreciate there’s a lot of bugs and QA is working hard.

But don’t add any more features while there’s even 1 remaining bug. Finish all the bugs before any new features. All of them.

One thing I noticed is that when I submit a report I have no idea if Unity QA tried to reproduce it and was not able to. I can handle being told my bug is not reproducible :) I just don’t like not knowing if it was ever looked at, this made me stop submitting bugs that require specific steps to reproduce.

Hey! Thanks, interesting, I appreciate it. Tomorrow I give a short talk at CodeMotion Berlin about game development with Unity. It suits. I’ll probably show it in order to explain some game dev issues.

@Klemens Forster: there is already support for Ouya. It’s called Android and Ouya have a free pack with some helpful tools for Ouya.

I would like to know why you people have managed to illegally sneak your software into my Hard Drive without my Permission No wonder my Computer is slowing down it is Third party software like you sneaking into people’s Hard drives that really pushes my nerves.

Do it again and i will file a Report with the Authorities. Plain and Simple.

Thanks for sharing! I try and read all of the QA blog posts :)

@ImaginaryHuman There will always be bugs in Unity, that’s just the nature of having a large codebase. I do know that they’re working hard on getting those bugs fixed as fast as possible as no sane developer wants to have any bugs in their software, so you can be sure they want to fix them :)

It’s be really awesome if you guys could let us know what planned features are coming in 4.2, this whole “ohh it’s a seecret” only really works for games. I’d specifically like to know about the shader editor and garbage collection optimizations.

You didn’t have to black out “PlayStation 4” and “Xbox Fusion”. It’s okay. Everyone already knows.

It’s just great that you put such an honest status reports of your code base. It means you trust the public which is scary and hard and great of course and means you are confident in your team and product which is again great. Congratulation for such a great strategy.

Awesome, thanks for the insight into the process. But of course now I want to know what’s under the blackouts.

Comments are closed.