Categories & Tags
Archive

Bug Reports, Incidents and some bashing

October 28, 2013 in Community News and Info by

When a bug report is sent to us, it becomes a “QA Incoming Incident” (see http://blogs.unity3d.com/2012/08/01/i-like-numbers/) that QA will have to process. At least we process as many as we can, because we receive A LOT of them (~1100 per week). We have been unable to even respond to a fraction of them, because we always prioritize our latest version (beta and alpha) higher. We do that because the chance of getting those bugs fixed is higher and thus more valuable to the product. We really want to respond to everyone, but it has to make sense for us and to figure out if it really does, we had to get some data.

We wanted answers to a few questions:

  • How many bugs can we find in those reports?
  • How good is our rating system? (brief explanation below)
  • How much time should we invest in them?
  • How much information can we get by asking users for more info on a report?
  • Can we somehow improve the workflow for everyone, including users?

The rating system

The rating system is a way for us to categorize incidents based on how good the report is, i.e., how likely is it to be reproducible and an actual bug in the system. To do this, we look at how much information is on the bug in the form of attachments and how much text has been entered in the description.

  • 0: Very little or no description. No attached files. User selected “First time” or “Sometimes”.
  • 1: Very little or no description. No attached files. User selected “Always”
  • 2: A small amount of description.
  • 3: An attachment is present (typically a project), but little to no description.
  • 4: A good amount of description is present, but no attachment.
  • 5: Good description and attachments present.

It is a very simple and crude rating, so it was interesting to see some data on whether it really worked.

Enter the Incident Bash

We had at the time, Tuesday October 15th, ~2000 active QA Incoming Incidents on version 4.2.1f4. Of these I randomly selected 25%, resulting in 505 reports we wanted to process. All hands on deck for one day. 38 people doing whatever we could to get this to 0 in one day. And we made it.

IncidentBash1

The initial 505 incidents were spread on these ratings:

IncidentBash2

More than 60% were in the first three ratings, meaning little to no input from the user. Only 60 had a good description.

After the incident bash, we looked at the number of bugs we had reproduced:


IncidentBash3

So after the initial processing we discovered 5.94% reproducible bugs in the pile of 505 incidents.

IncidentBash4

Of the incidents we processed, we replied to the user on 185 of them asking for more information about the bug; 37 were duplicates of existing bugs; 72 did not contain enough info to be reproducible nor had an email been sent to ask for more information, and finally, we responded to the user in 181 cases explaining that the incident was not caused by a bug, but by something else going wrong.

Of the 185 cases where we asked for more info we received an answer from 57 of the users one week later. The total bug-count had risen by 2. The final bug count total is:

IncidentBash5

That number is not exactly awesome. Let’s see how the rating distribution was:

IncidentBash6

Combine that with the total number of cases in each rating and we get the following probabilities of an incident turning into a bug based on the rating:

IncidentBash7

So the answer to the question of whether our rating system works is a big yes!

Let’s compare this to our alpha and beta users. Remember how I said we focus a lot more energy on those reports?

IncidentBash8

As you see, the probability of getting a bug out of a report from our beta users is higher. It makes sense for us to focus a lot more energy on those reports than on the reports from the publicly released versions.  Also, more than 80% of all bugs in the alpha/beta groups are rated 3-5.

The distribution of the bugs we found was without large spikes. That is a good sign of not having great holes in our system:

IncidentBash9

We had 3 regressions in that pack of reports, which is a nice find, but not scary to us.

Conclusions

First of all I have a plea: Dear user, if you want to increase the chance of a bug you report being reproduced and fixed, give us a good description of the problem and attach a project or video to the report. Based on the small experiment we did, providing such information results in a higher chance that we will be able to reproduce the bug (2-44x higher, depending on which extremes you compare). Reproducing the bug is the first step to even considering fixing it.

Next, we have a well functioning rating system. We want to incorporate this better in the future versions of the bug reporter.

Finally, I want to say one thing loud and clear: If you get a crash in the editor, press “Send Error Report” even if you have no clue what happened! We have a separate system for automated analysis of crash reports, which we use for finding bugs from pure statistics on the crashes happening. That system is worthy of a separate blog post, so stay tuned for that.

Share this post

Comments (27)

Comments are closed.

Joel
28 Oct 2013, 6:45 pm

Great breakdown!

28 Oct 2013, 7:23 pm

This was a good read and it makes totally sense to have some sort of “spam filter”.

However, it also feels like when a bug-report contains a rather detailed and more complex description/reproduce-steps, it also takes much longer until it gets processed by the QA.

I submitted a few issues in the past. All containing detailed reproduce steps along with a sample project, where I took the time to create a project that exactly shows the problem in a 100% reliable way to reproduce the issue. The QA is usually quite fast, responding to those reports within hours to days, but the more complex issues don’t get processed that fast (still waiting for a response though).

Bug-reports where I wait for a response:
#563755: ShaderInspector displays wrong LOD value (1 month ago)
#560816: Changing a *.cginc file forces reimport on all shaders (2 months ago)

Elvis
28 Oct 2013, 7:37 pm

@PETER77 Thank you for taking the time to send us bug reports with detailed repro steps and a project attached. The reason why you didn’t receive a response from QA on those two cases you mentioned is that they were submitted with Unity 4.1.2. We can barely keep up with looking at and responding to cases submitted by our alpha and beta testers and by users on the latest released Unity version, namely 4.2.

We might get to your reports and all the others submitted in Unity 4.1 and earlier, but I can’t make any promises. From this blog post you can see how easily a good bug report like yours can get drowned in the incoming sea of fairly bad bug reports. We would need an army of QA people to go through each and one of them.

However, all is not lost. We are trying to get smarter every day about how we handle bugs that are reported to us. This experiment was just an example of what we are trying to do. We also have plans of making parts of the bug database public to our users and we have the crash analysis tools which we might talk about more in a future blog post.

For now, all I can say is, keep submitting those bugs, try to update to the latest released version of Unity and stay awesome!

Elvis
QA Lead

28 Oct 2013, 7:41 pm

@Elvis: Thanks for the response! I was pretty sure the delay is due to me using 4.1 on those bug-reports. That’s why I recently installed a newer version in parallel, just to submit issues with it, to get a better QA rank, heh .-)

Stay awesome to you too!

Thomas Petersen
28 Oct 2013, 9:27 pm

@PETER77: As long as the bug is also present on that later version, you are very welcome to submit the bug on it. However, if you submit a bug on 4.2.2 which is only present on 4.1.5, you will obviously be wasting a lot of time since we won’t be able to reproduce the bug on that version.

Sebas
29 Oct 2013, 1:03 am

what about this bug:

case 555189 (duplicated 555200). Has been reported by several users on the forum already. Also, is there a way to see the reported bugs?

Troy
29 Oct 2013, 5:14 am

I appreciate the approach to problem solving.. the more data we can provide, the higher the likelihood it will be addressed.

thanks for this!

Julien
29 Oct 2013, 6:52 am

Thanks Elvis – I love that a) you guys are trying to improve, and b) you’re doing it by looking at the data.

It’s always been a pleasure to work with the QA team.

29 Oct 2013, 8:08 am

@Thomas Petersen: If an issue doesn’t occur in the version used to submit the bug-report, there is no need to submit it at all, obviously. People around you are not brain-dead, thanks for the response anyway.

jcr
29 Oct 2013, 9:19 am

Have you considered that the reason you are more likely to get bugs from alpha and beta users is that alpha and beta versions inherently contain more bugs? How many label 0 bugs are actually duplicates of label 4 or 5 bugs in beta builds? It seems a better strategy would be to just focus on label 4 – 5 bugs, occasionally going to label 3, regardless of whether they are filed against a beta or release build.

29 Oct 2013, 9:21 am

I recently found out that you tend to ignore bugs from older versions, which was a shock to me, because it meant that most (all?) of my bugs will be ignored. And I invested time in sending good reports. The most disappointing part is that I didn’t know that that will happen. The simple solution to that: add display of bug rating in the “Submit a bug” dialog (and explain the rating there). That way the user will know what kind of extra things you need from us.

I also think that testing only recent version bugs tends to find only easier bugs, because big projects tend to avoid changing of Unity version, which means that you ignore bugs which report serious usability issues.

The last thing which I mentioned to some guys at Unity: why don’t you have rating based on a user, i.e. on the previous bug submissions? I think that criteria on it’s own would be more effective than what you have now. Sure you don’t want to base it purely on that, but some mix would work very well.

Riley
29 Oct 2013, 10:55 am

Is there a way to see if a bug has already been reported, a tracker of sorts?

Thomas Petersen
29 Oct 2013, 11:06 am

@Peter77: Never have thought they were.

@JCR: There is some sense to that, yes. However, we get a tsunami of incidents from the public versions, yet very little bugs. The exact opposite is the case for the alpha/beta groups. We also have a distribution from the groups of 76.25% landing in rating 3-5, while the exact opposite is happening from the public. And yes, usually we are taking the highest rated bugs first, this was merely an exercise in obtaining data from the entire dataset.

@Paulius: Yes, we are ignoring versions older than latest public. They may be harder, but they are also several thousand bugfixes and changes behind, so in order to capture the bug, we would have to verify it on, say 4.1.5, then figure out if it was reproducible on the latest release branch and only then actually make it a real bug. It is unfortunately a matter of not having all the time in the world to handle everything and thus we prioritize. Maybe we could take some opportunistic approach to older versions, but we will never really dig into them.

Thomas Petersen
29 Oct 2013, 11:07 am

@Riley: No, not yet. But there will be. Soon (TM).

Peter Dwyer
29 Oct 2013, 4:31 pm

Sadly in my years of experience with Unity. You guys tend to ignore bugs that you simply don’t want to fix, regardless of if they are priority one or not.

Take the windows RT keyboard not being accessible or the mic not working. Apparently, though these are vital things for a lot of apps (enter your name being something most apps do ) and has been an issue since 4.2.0. It’s not actually of any interest to bother to fix it. Reminds me of those agency companies that loose all interest in fixing their code once they’ve been paid!

Don’t get me wrong. I love unity but, sometimes it feels like the guys and gals there only do the stuff you feel is fun, like Unity2D and forget to fix the annoying bugs we see every day and actually need fixing. I can’t for instance even play an intro video in windows store apps without resorting to xaml!

I know this kind of stuff is boring but, if you are going to support a platform at least do it to the standard of the others like iOS and Android. Otherwise what’s the point?

29 Oct 2013, 6:00 pm

Thank you for the information about how a bug is rated! I’m especially pleased to find out that good user information is valued above the sending of a project.

Is it possible to create something automatic on your end that will return an automatic message to a user who has submitted a bug in a version that is not the latest, telling them to upgrade? Then, at worst, a player will only submit an unusable bug once. Even better, disable the bug-sending form for a previous version of the editor (not sure if that’s possible, though).

Either way, it sounds like there has been a lot of wasted user effort into writing bug reports that you’ll never read.

Thomas Petersen
29 Oct 2013, 6:25 pm

@Chris: We are looking into doing some changes, but changing the bug reporter is super high risk for us, so it will take time before it lands in users hands.

30 Oct 2013, 12:18 am

@Thomas: Yeah, that does sound like a particularly risky thing to do. If you got it wrong, how would we let you know? :)

UnityUser
30 Oct 2013, 1:32 pm

I agree with Paulius. If you wish to make unity bug reporting process more efficient:
1. Show in the bug reporting dialog the rating of the bug about to be submitted, with explanation on the value shown.
2. If the Unity version used is not the latest, gray out the ‘submit’ button and explain there that bugs needs to be reproduced and submitted from the latest version of Unity to be processed.

Otherwise, your time and the users time is wasted.
In addition, users who don’t read your blog posts and not aware of your rating system, will eventually find it useless to submit bugs, and that the product is not well supported.

Wel
30 Oct 2013, 6:09 pm

Keep up the good work, I really hope Unity become the best engine ever.

neo
31 Oct 2013, 11:34 am

Thanks for the great tutorials but please add more specially for advanced scripting and effects like particles.

UNITYUSER
4 Nov 2013, 8:24 pm

I just had Unity crashed, without me doing anything special.
It brought up the bug report dialog.

Now, what I’d expect from a decent software company, is to inspect this crash given the log files, exceptions thrown, stack data etc.

But given this article, having no relevant data in the project to attach, having this crash happen in the first time, and nothing much to describe on it, I know that Unity developers will never even have a look at this bug.

Great plan guys (sarcasm)
Keep on having thousands of bugs in your DB, you deserve each of them.
I wouldn’t be surprised of most of them are the same bugs being sent over and over again as you never fix them for these exact reasons.

Sure, every developer would be happy to fix bug that are 100% reproducible, given a tiny project to reproduce and fix it with.

Maybe you should learn from projects like TortoiseSVN how to bug a product efficiently and properly. They had a blog post exactly on that.
And they don’t ask anyone to attach their entire repository or create one to help them reproduce bugs.

Thomas Petersen
4 Nov 2013, 8:29 pm

@Unityuser: “Finally, I want to say one thing loud and clear: If you get a crash in the editor, press “Send Error Report” even if you have no clue what happened! We have a separate system for automated analysis of crash reports, which we use for finding bugs from pure statistics on the crashes happening. That system is worthy of a separate blog post, so stay tuned for that.”

So just press send.

4 Nov 2013, 11:29 pm

A few ideas:
- Can you improve the tools to report bugs? If you like screenshots can you add an option for paused game, send bug report w/ including the current frame? How about the editor?
- Can you weight attached files? For example, sending in a small amount of files probably indicates the user put more effort into reproducing the bug rather than sending an entire project?
- Do you tie accounts to bug reports? If so, can you create a heuristic per-account? Let’s say someone sends in reliable bug reports, can they be bumped into a higher bucket?
- While I understand your desire to prioritize beta bugs, long-standing bugs probably affect more users.

UNITYUSER
6 Nov 2013, 7:30 am

@Thomas Petersen
My mistake, I read this post a few days earlier and that part slipped away from my memory.

With regards to submitting crash reports, you shouldn’t be asking people to submit their project. It’s unreasonable. As mentioned in the last paragraph, it can be done differently.
I found the post in TortoiseSVN website:
http://tortoisesvn.net/crashhandler.html

With regards to non-crash bugs, since Unity users are basically developers, it would make sense to explain that attaching a small sample project showing the problem with reproduction steps is obligatory, and disable the submit button otherwise.

This may have some users avoid submitting “broken functionality” bug reports, but would reduce the number of incoming bugs to something you can deal with.
It doesn’t make sense to constantly have more bugs than you could ever handle.

I’m sure that this suggested change to the bug reporter interface won’t take more than a few hours to implement, and will improve your effectiveness in dealing with bugs in the long run.

MG
18 Dec 2013, 1:57 pm

is there a chance for us freelancers to help resolve the bugs?

like finding if it really is a bug and so on?

I mean 35 is very little number to process this all and we are in thousants of numbers, if just few of us and I believe allot will help it would speed the process allot, …

OFC we would do it for free, …

Thomas Petersen
18 Dec 2013, 2:04 pm

@MG: Maybe some day, but for now we have the public bug site issuetracker.unity3d.com where you can find confirmed bugs. We are very worried about privacy and there are a few work flow problems to solve before we can make incidents public.

Leave a Reply

Comments are closed.