Search Unity

It’s been 1.5 years since we last did an incident bash, so time was ripe for executing another in order to get some fresh data. Enter the Great Incident Bash of 2015.

Just to make sure we have the terms in place: An incident is when a user sends us a report through the bugreporter. An incident is turned into a bug when one of our testers has successfully reproduced the incident as a bug.

An incident bash is a period of time where we mark up a random sample of bugs within a given time period. In this case, we took 28 days of incidents that came in on version 5.0.1f1. We looked at 5.0.1f1 because it is the first release to roll up the initial patches after 5.0. We get a majority of incidents a month or two after the release, so this was a good time to look at it. Our sample had a total of 2058 incidents of which we randomly marked up 25%, leading us to fully investigate 491 cases.

Fully investigating a case means that we try to reproduce every case, respond to the user if we miss information, follow up on every response and try to get to the bottom of each case. This is immensely time consuming, but doing it on a random sample gives us a lot of information.

Going through 491 incidents, we managed to turn 73 into a bug:


That’s a total of 14.86% of the entire population. Compared to the previous value of 6.83%, this is a lot more valid bugs, but it is also very low compared to the effort of digging them out. In other words, 85% of all incidents we get are in fact not bugs or duplicates of known issues. Having to wade through 7 bugs to actually find something real is an immense waste of time, so normally we don’t do that.

What we do is prioritize incidents according to a rating system. The rating takes into account the description, attached projects, images and files. It ranges from 0 (crash, profanity, no added information) to 5 (project attached and a solid description). We also handle incidents from our enterprise support customers first, because we have direct relations with them already. As such, we take incidents in the following order:

Enterprise customers -> Pro Rating 5 -> Rating 5 -> Pro Rating 4 -> Rating 4 -> etc.

Because of the incident bash, we now have the data to check whether this system is working.

The overview of the ratings on the incidents we started with shows that 4 and 5 are the smallest portion of the total pool.image00

We can see that 55 Rating 5 and 35 Rating 4 incidents out of the total 491 is not a lot. The one incident without a rating got through by mistake, but we included it for completeness sake.

Looking at the spread of the bugs according to the rating they had, we see the following picture:


As expected, the vast majority of actual bugs were those with a rating of 5. And that’s out of the smallest pool of incidents.

The rating 0 incidents that do turn into bugs are most likely crashes that we have automatically identified through our crash analysis tool. We will blog about that later in the year. For now, please make sure you send all crash reports to us, even if you don’t add information. We can use the data for processing automatically.

Rating   Chance of Bug
0 1%
1 0%
2 16%
3 14%
4 20%
5 62%

This table really tells the story of why we are so insistent on getting a repro project attached and get a good description as well. Rating 4 is a solid description, but no attachment, while 5 is both. The difference is a 3 times larger chance of reproducing the issue and ultimately filing a bug that can be fixed.

It also tells the story of why we prioritize rating 5 so high. We are hunting bugs, as many as we can in the time we have available. That means prioritizing the effort on where the biggest chance of finding a bug is present. We will never be able to handle and respond to all incidents, there are simply too many incoming. To date, we have gotten 4.510 incidents reported on just version 5.0.1f1, of which only 10.7% percent have rating 5.

There are other things we are working on to change the flow of incidents. Coming in 5.3, we will integrate an automated search in the bug reporter itself. This will attempt to present possible solutions to a user as they are filling out the form, which will hopefully allow them to solve the problem much faster than having a turnaround with our team.

Further down the road, we will integrate crash lookups with our internal crash analysis tools, so a crash will automatically be able to tell the user whether it is a known problem, if there is a patch that solves it or if this is a new crash and we need more information. There are many other ways we can improve the bug reporter and we’re continuously working on it.

24 replies on “The Great Incident Bash of 2015”

Theme list option in Administration panel sholud use Stylesheet instead of Template.~ Line 223echo .$theme[“Name”]. ;This is because some themes have templates (parents) and therefore child themes can’t be selected on the admin panel. Of course other references to the same value sholud also be changed.

Great post. Have you also thought about rating bug reporters? I know systems like this from release engineering when merging contributions. Users could start with 3 of 5 stars. If they report incidents that turn into bugs they increase their “trust” score otherwise it decreases. A 5 star user could be trusted to report important information even if the report is only rated 2 or 3.

Great insights into your QA internals, thanks.

We have a very large project codebase here. Although I try to simplify and isolate bugs before reporting, sometimes its just not so easy. We got a couple of crashes and/or problems that are perfectly reproducible with our large codebase (~20 gigs) but it is totally out of the question to upload a copy of that project every time we submit a bug report.

We already sent access to a git snapshot clone to Rene Damm some weeks ago. If you sent us some pub-key, we can also arrange access to the latest git repo. Is it possible for Unity bug reports to refer to a project by “checkout hash #abc123 from our GIT xyz that we gave you some time ago” instead of attaching the whole project every time? As I understand the posting, this would not qualify for a “Level 5 bug report”. (We could also do it the other way round and check-in into some branch of a repo hosted at Unity – if thats easier for you.)

Also, we often report bugs with our working version of unity – usually a couple of months old. I always check them some days later whether they are still present in the latest Beta (so we might have a reason to upgrade ^^). I usually add some line “Still present in X.Y.ZbW” to the bug report. As I read this blog, this won’t raise the quality level of said bug report, right? Should there be some structured way of updating the version in which the bug still appears?

Hi Immanuel,

It is OK to send bug reports where you are referring to a project you have already shared with someone at Unity (though I suggest you also mention the name of the person you shared the project with, so that whoever is looking at your bug report knows who to contact to get the project details).

You are right in that by doing this the rating of the bug is going to be lower and that by adding more details to the bug later on will not increase the rating (something for us to consider). However, making updates to your reports is still welcomed since when someone will get to look at it, there will be as much information in there as possible.

Thomas already described some of the improvements we are adding to the bug reporting tool and rest assured that we will continue improving on our handling of all incoming reports for the future as well (keep an eye on this space for when we announce more details about our crash analysis tool, for example, that will be pretty awesome)

One thing I’ve found useful is, when you find a bug, to create a new empty project and attempt to reproduce it in the most minimal way possible. The reason for this is twofold:

1.) You may, in fact, find that you did something wrong and it is not a bug at all. This happens to me a lot.
2.) Once you have a small repro case, you can attach the whole project to the bug report and it will significantly reduce both the time and bandwidth it takes to send the report, and gives the Unity team a nice small package without all the cruft of your whole game project.

Nice post.

Unfortunately (for our team), the response from QA regarding most of the bugs i have submitted lately are pretty poor (i believe i provide enough information in the description, as always).

Most of the issues i raised did not receive any response. I recall things to be different a while ago. Perhaps it has to do with the current release phases and what not.

A few years ago I sent a bug report for a problem that was only reproducible on an Intel GFX chip that Unity had already said would be unsupported in the next release (3 – 3.5 – 4, I can’t remember which it was) and they not only reproduced the bug but fixed it in time for the text minor release less than 7 days later.

From that moment on as soon as I saw a bug I thought could be in the engine or editor I’d spend 15 mins trying to create the smallest project to show it in effect to submit a bug report on.

By acting fast and taking the time to fix a bug that was only ever going to effect a small portion of their developers and their final customers they turned me into an unofficial part of the QA team :)

Those bug reports really to get acted upon people.

I always try my best for bug reports, to help the recipient save time, but also to make sure I know how to reproduce the problem exactly with an example as solid proof. By doing that, I also make sure I’m not just misusing the software, sometimes I learn that it was my mistake, just by writing the description. Time and effort saved for everyone!

The only bug report I filled for Unity was a 5 I guess, from what you say in this post, and I got an answer only 24h later. It said the bug is real but low priority.

Very interesting.

I wonder if you can give a bit more information to get a 5 point rating to the bug report (when is the description good and when is it enough to submit a small test project).

I’m asking because I’ve got some tickets (with descriptions and test projects), but I get hardly feedback to them (mostly being in a monologue). Don’t understand me wrong: I think the QA process has improved a lot from you guys in the last 1-2 years, but I find it hard to understand when a ticket is helping you to find the bug and when not.

I want to produce quality bug tickets, so you have an easier time to find the problem and solve it (which would make me happy, ’cause they are affecting me and my project). If I create a bug report and invest time into it, it’s a bit frustrating to see some of my tickets going into the void (or at least it seems to me).

So, transparceny is a good thing! :)

Comments are closed.