Search Unity

It’s been 1.5 years since we last did an incident bash, so time was ripe for executing another in order to get some fresh data. Enter the Great Incident Bash of 2015.

Just to make sure we have the terms in place: An incident is when a user sends us a report through the bugreporter. An incident is turned into a bug when one of our testers has successfully reproduced the incident as a bug.

An incident bash is a period of time where we mark up a random sample of bugs within a given time period. In this case, we took 28 days of incidents that came in on version 5.0.1f1. We looked at 5.0.1f1 because it is the first release to roll up the initial patches after 5.0. We get a majority of incidents a month or two after the release, so this was a good time to look at it. Our sample had a total of 2058 incidents of which we randomly marked up 25%, leading us to fully investigate 491 cases.

Fully investigating a case means that we try to reproduce every case, respond to the user if we miss information, follow up on every response and try to get to the bottom of each case. This is immensely time consuming, but doing it on a random sample gives us a lot of information.

Going through 491 incidents, we managed to turn 73 into a bug:

image02

That’s a total of 14.86% of the entire population. Compared to the previous value of 6.83%, this is a lot more valid bugs, but it is also very low compared to the effort of digging them out. In other words, 85% of all incidents we get are in fact not bugs or duplicates of known issues. Having to wade through 7 bugs to actually find something real is an immense waste of time, so normally we don’t do that.

What we do is prioritize incidents according to a rating system. The rating takes into account the description, attached projects, images and files. It ranges from 0 (crash, profanity, no added information) to 5 (project attached and a solid description). We also handle incidents from our enterprise support customers first, because we have direct relations with them already. As such, we take incidents in the following order:

Enterprise customers -> Pro Rating 5 -> Rating 5 -> Pro Rating 4 -> Rating 4 -> etc.

Because of the incident bash, we now have the data to check whether this system is working.

The overview of the ratings on the incidents we started with shows that 4 and 5 are the smallest portion of the total pool.image00

We can see that 55 Rating 5 and 35 Rating 4 incidents out of the total 491 is not a lot. The one incident without a rating got through by mistake, but we included it for completeness sake.

Looking at the spread of the bugs according to the rating they had, we see the following picture:

image01

As expected, the vast majority of actual bugs were those with a rating of 5. And that’s out of the smallest pool of incidents.

The rating 0 incidents that do turn into bugs are most likely crashes that we have automatically identified through our crash analysis tool. We will blog about that later in the year. For now, please make sure you send all crash reports to us, even if you don’t add information. We can use the data for processing automatically.

Rating   Chance of Bug
0 1%
1 0%
2 16%
3 14%
4 20%
5 62%

This table really tells the story of why we are so insistent on getting a repro project attached and get a good description as well. Rating 4 is a solid description, but no attachment, while 5 is both. The difference is a 3 times larger chance of reproducing the issue and ultimately filing a bug that can be fixed.

It also tells the story of why we prioritize rating 5 so high. We are hunting bugs, as many as we can in the time we have available. That means prioritizing the effort on where the biggest chance of finding a bug is present. We will never be able to handle and respond to all incidents, there are simply too many incoming. To date, we have gotten 4.510 incidents reported on just version 5.0.1f1, of which only 10.7% percent have rating 5.

There are other things we are working on to change the flow of incidents. Coming in 5.3, we will integrate an automated search in the bug reporter itself. This will attempt to present possible solutions to a user as they are filling out the form, which will hopefully allow them to solve the problem much faster than having a turnaround with our team.

Further down the road, we will integrate crash lookups with our internal crash analysis tools, so a crash will automatically be able to tell the user whether it is a known problem, if there is a patch that solves it or if this is a new crash and we need more information. There are many other ways we can improve the bug reporter and we’re continuously working on it.

24 Comments

Subscribe to comments

Comments are closed.

  1. Theme list option in Administration panel sholud use Stylesheet instead of Template.~ Line 223echo .$theme[“Name”]. ;This is because some themes have templates (parents) and therefore child themes can’t be selected on the admin panel. Of course other references to the same value sholud also be changed.

  2. That’s a great article, It’s always reassuring when things are breakdown with graphs and such ;)
    Thanks!

  3. Great post. Have you also thought about rating bug reporters? I know systems like this from release engineering when merging contributions. Users could start with 3 of 5 stars. If they report incidents that turn into bugs they increase their “trust” score otherwise it decreases. A 5 star user could be trusted to report important information even if the report is only rated 2 or 3.

  4. vitor alexandre colomby

    August 22, 2015 at 12:25 am

    Contract Wars

    1. vitor alexandre colomby

      August 22, 2015 at 12:27 am

      battlefield 5

  5. Immanuel Scholz

    August 18, 2015 at 10:37 am

    Great insights into your QA internals, thanks.

    We have a very large project codebase here. Although I try to simplify and isolate bugs before reporting, sometimes its just not so easy. We got a couple of crashes and/or problems that are perfectly reproducible with our large codebase (~20 gigs) but it is totally out of the question to upload a copy of that project every time we submit a bug report.

    We already sent access to a git snapshot clone to Rene Damm some weeks ago. If you sent us some pub-key, we can also arrange access to the latest git repo. Is it possible for Unity bug reports to refer to a project by “checkout hash #abc123 from our GIT xyz that we gave you some time ago” instead of attaching the whole project every time? As I understand the posting, this would not qualify for a “Level 5 bug report”. (We could also do it the other way round and check-in into some branch of a repo hosted at Unity – if thats easier for you.)

    Also, we often report bugs with our working version of unity – usually a couple of months old. I always check them some days later whether they are still present in the latest Beta (so we might have a reason to upgrade ^^). I usually add some line “Still present in X.Y.ZbW” to the bug report. As I read this blog, this won’t raise the quality level of said bug report, right? Should there be some structured way of updating the version in which the bug still appears?

    1. Hi Immanuel,

      It is OK to send bug reports where you are referring to a project you have already shared with someone at Unity (though I suggest you also mention the name of the person you shared the project with, so that whoever is looking at your bug report knows who to contact to get the project details).

      You are right in that by doing this the rating of the bug is going to be lower and that by adding more details to the bug later on will not increase the rating (something for us to consider). However, making updates to your reports is still welcomed since when someone will get to look at it, there will be as much information in there as possible.

      Thomas already described some of the improvements we are adding to the bug reporting tool and rest assured that we will continue improving on our handling of all incoming reports for the future as well (keep an eye on this space for when we announce more details about our crash analysis tool, for example, that will be pretty awesome)

  6. One thing I’ve found useful is, when you find a bug, to create a new empty project and attempt to reproduce it in the most minimal way possible. The reason for this is twofold:

    1.) You may, in fact, find that you did something wrong and it is not a bug at all. This happens to me a lot.
    or
    2.) Once you have a small repro case, you can attach the whole project to the bug report and it will significantly reduce both the time and bandwidth it takes to send the report, and gives the Unity team a nice small package without all the cruft of your whole game project.

  7. Nice post.

    Unfortunately (for our team), the response from QA regarding most of the bugs i have submitted lately are pretty poor (i believe i provide enough information in the description, as always).

    Most of the issues i raised did not receive any response. I recall things to be different a while ago. Perhaps it has to do with the current release phases and what not.

  8. It would be really nice to get an explanation when issue is marked as Won’t Fix.
    e.g. http://issuetracker.unity3d.com/issues/android-several-unnecessary-permissions-are-added-to-app-manifest

    1. Thomas Petersen

      August 18, 2015 at 12:22 am

      I agree. The description should have been updated with this information.

  9. Hi Thomas, and thank you for shedding some light on a topic that interested me for a while, how it works.

    I am curious, what do you do about bugs like these?

    Game Center achievement not working: http://forum.unity3d.com/threads/problem-with-game-center-achievements.310817/

    I have reported it a few times, there is a (now mature) forum thread.

    What is some other way to bring this to QA’s attention (or whomever it concerns)?

    Thank you!

    1. Thomas Petersen

      August 18, 2015 at 12:21 am

      I think you did the right thing by posting on the forums. Helped out a lot of people.

      The incident in the thread had not been looked at. Why that is the case, I can’t say, but it was rated 4. There’s a good description on it, but no project attached, which is why it is going one down.

      Anyways, it looks interesting and we’re going to look into it.

  10. I wrote a post last month about how I think the bug tracker could be improved:
    http://forum.unity3d.com/threads/improved-bug-tracker.339570/

    The automated search and crash analysis lookup sound good, hopefully you’ll be adding some interaction possibilities too.

    1. Thomas Petersen

      August 18, 2015 at 12:03 am

      The post has a lot of good suggestions, of which many are along the same lines we are thinking. It will take a while, though, because we have to integrate our backend systems with the login in the editor and then we can start playing with the data.

      But this takes time. Longer than you would expect, because this train can’t stop for just a single day, so we must be extremely careful.

  11. A few years ago I sent a bug report for a problem that was only reproducible on an Intel GFX chip that Unity had already said would be unsupported in the next release (3 – 3.5 – 4, I can’t remember which it was) and they not only reproduced the bug but fixed it in time for the text minor release less than 7 days later.

    From that moment on as soon as I saw a bug I thought could be in the engine or editor I’d spend 15 mins trying to create the smallest project to show it in effect to submit a bug report on.

    By acting fast and taking the time to fix a bug that was only ever going to effect a small portion of their developers and their final customers they turned me into an unofficial part of the QA team :)

    Those bug reports really to get acted upon people.

  12. Arnaud Couturier

    August 17, 2015 at 3:11 pm

    I always try my best for bug reports, to help the recipient save time, but also to make sure I know how to reproduce the problem exactly with an example as solid proof. By doing that, I also make sure I’m not just misusing the software, sometimes I learn that it was my mistake, just by writing the description. Time and effort saved for everyone!

    The only bug report I filled for Unity was a 5 I guess, from what you say in this post, and I got an answer only 24h later. It said the bug is real but low priority.

    1. Arnaud Couturier

      August 17, 2015 at 3:12 pm

      … forgot to add: and Unity staff also provided an easy workaround, so good job, and thank you guys!

  13. Very interesting.

    I wonder if you can give a bit more information to get a 5 point rating to the bug report (when is the description good and when is it enough to submit a small test project).

    I’m asking because I’ve got some tickets (with descriptions and test projects), but I get hardly feedback to them (mostly being in a monologue). Don’t understand me wrong: I think the QA process has improved a lot from you guys in the last 1-2 years, but I find it hard to understand when a ticket is helping you to find the bug and when not.

    I want to produce quality bug tickets, so you have an easier time to find the problem and solve it (which would make me happy, ’cause they are affecting me and my project). If I create a bug report and invest time into it, it’s a bit frustrating to see some of my tickets going into the void (or at least it seems to me).

    So, transparceny is a good thing! :)

    1. Thomas Petersen

      August 17, 2015 at 5:02 pm

      First of all, thanks for submitting the bugs, Pahe.

      If you have provided a description and a attached a project to your bug report, that would get it high on our priority.

      What I didn’t mention was that we also take the version number into consideration, so older versions get less and less attention. And when I mean “older”, it can be only a few months old, because that means the codebase has moved SO much in the meantime that we can be chasing a bug which is no longer valid. To give this some context, the Unity codebase gets more than 4500 changesets committed every month, so when the latest official release is 5.1.2f1, that codebase is over 9000 (yes, I said it) changesets behind the one we are currently working on in 5.2. So, submitting reports on the latest possible version is another good practice in order to get the incident prioritized.

      1. Is this per major version? I.e. the latest 5.x release and the latest 4.x release? I understand that 5.x gets the most love, but I’m wondering where 4.x fits in, considering some of us are stuck in 4.x for a while (in our case to support iOS 5.1.1)

        1. Thomas Petersen

          August 17, 2015 at 11:59 pm

          4.x only gets attention if we have issues from our enterprise customers OR there is an issue with IL2CPP. In some cases we also back port a fix all the way to 4.6, but it is getting very rare. 4.x will sunset completely by the end of this year.

      2. So lets say I submitted a bug for 5.1.2 (with repro and description) and it didn’t get a response. And the bug still exists in 5.2. What is a good way to make sure a bug that’s been around for a long time still gets attention?
        Submitting another duplicate bug with 5.2 doesn’t seem productive for anyone involved.

        1. Thomas Petersen

          August 17, 2015 at 11:57 pm

          You’re right, it doesn’t. We will focus on 5.1.2, but if you were to submit a bug for 5.1.0f1, it has a very high risk of not getting attention, especially if it is also rated below 5.

          As you can tell, there are a ton of different priorities at work, so the best pointer I can give is to be on the latest version. Patch versions if they apply to you, but otherwise latest version updated through the editor.