Search Unity

My name is Igor and I am a Toolsmith at Unity, which means I am part of the team that build tools to increase productivity of Devs and QA in Unity with the aim to improve the overall quality of the product and the experience of our Users.

What was there before…

Let’s start with a little bit of history of handling bugs in Unity. There is a tool installed with the Editor called the Bug Reporter, which could be launched either manually or automatically in case of a crash (see more at Reporting-a-bug). After a user submits a bug report, QA has to try to reproduce the issue and turn this report (which is initially called an incident) into a bug, which will be passed to development teams for triage and/or fix or other solution. Good reports should contain a descriptive title, steps to reproduce and have a project (ideally – a small one focused on the problem) attached to it, so it is easy for us to verify and reproduce the issue. That’s what we at Unity always hope to get in the report (more at Attaching-your-project-to-a-bug-report).

The problem

But here comes another side of the equation. Having more than a million registered users we receive A LOT of reports of the same bug, which is good, until it becomes bad: someone has to look at every report that gets sent (around 6000 per month), verify it, reply to the user and so on. And after we verified the bug (and/or even have a fix for it ready), all the additional reports sent aren’t helping to solve the problem but taking valuable time from QA going through them. Automation to the rescue!

The solution

It turns out that crashes are the perfect example of a problem which in most cases is identified by one characteristic common to all of them: the callstack of the crash (i.e. the sequence of function calls in the program code of the Editor which eventually lead to a crash – with the name of the crashed function on top of it). Which means, unlike many other Bugs, those kinds of problems are much easier to group together by the machine without any user intervention (there are exceptions to that rule, but more about that later).

Benefits

When we started this project a few years ago we had no idea how many additional insights it would give us. From historical data of all the crashes across different versions of Editor (quantity, dynamics, etc), to the ability to immediately identify if a crash happened on a user’s machine already has a fix in a Unity version.

We built a tool which analyzes all the reports sent by users, parses all logs from the Editor attached to them to find the callstack of a crash and then maps identical or similar crashes together (figure 1). That gave tremendous value and increase in productivity for both developers fixing the issue (who now have all the similar reports at their fingertips and can quickly look for more information or other repro project) and testers (who can immediately see if the reported issue falls into a certain category and if there might already be a solution for it or at least a verified bug with a public Issue Tracker item for it where users can keep track of the issue, therefore providing the user with help in a timely fashion). Now release managers can also assess the stability and production readiness of the builds way before they make their way into alpha or beta testing, let alone stable releases (Unity Roadmap) and look for possible regressions and the User Pain caused.

Figure 1. Bucket page with similar crashes and reports statistics

Handling duplicate reports in Crash Analyzer

If for some type of crash we were able to turn one of the reports into a bug (i.e. have steps to reproduce / project provided by the user), we might want to close all other similar reports as duplicates (while providing users a link to track the progress on the bug fixing with Issue Tracker). What we can do is to mark the report as the repro for the crash and then resolve all the others as duplicates (figure 2).

Figure 2. Resolve & close duplicates if we already have a repro to work with

For everyone’s convenience Slack integrations were also added, so now if you want to receive notifications of new reported crashes (along with info showing if it is known or not, Unity version it was reported against, and so on) all you need is subscribe to a few channels (Figure 3).

Figure 3. Slack notifications when new crashes are reported

Known issues or things to address

We are not out of the woods yet! We keep working to improve our algorithms (some of the called functions on the stack are meaningless and should be filtered out, some of the platforms provide us with better callstack collecting mechanisms than others, etc). Some of the crashes are fully identified by their callstack and as a result, could be processed completely automatically. Sometimes callstacks must be 100% identical to be the same bug. Sometimes it is enough to be ‘similar’ (for example, have the same top frame – crashed function name, but varies down the stack a little bit). But in some cases even identical callstacks could mean different root causes. This often happens with the external calls into 3rd party libraries or drivers, where exact place of the crash itself is not enough and different parameters of the call or varying setup could result in different problems. For those crashes full automation is still not possible yet and it requires human investigation to tell the difference. The goal is to at least semi-automate cases like that, which means someone has to take a look, resolve possible issues manually before we can advance to automatic handling.  

More to come…

The plan is to further integrate with existing tools like Issue Tracker and Bug Reporter, so instead of collecting a user’s crash report, storing it at our servers, analyzing and then providing references to an existing bug and/or a solution for it, we’ll be able to exchange data right away between Bug Reporter and Crash Analyzer’s backend to prevent sending reports for already fixed issues and provide users with immediate feedback / solutions instead of submitting reports and waiting for a response from QA.

Stay tuned for more…

Comments are closed.

  1. It’s great, but I want to know this tool could we download to use or just send the problem to your team? I wish we could download to deal with our problem.

    1. It is internal only, you should submit your bug report as usual via Bug Reporter

  2. and also hopefully you notice that having the ability to easily visualize performance data (with PerfSheet for example) can be very helpful and time saving when trlosoebhouting non-trivial

  3. How do you submit bugs to the public issue tracker (https://issuetracker.unity3d.com/) ? I can vote for other’s bugs, but I see no way to submit them. And if I use Unity Editor’s bug reporter, it submits to the private fogbugz repository.

    1. Elena Savinova

      March 9, 2017 at 12:46 pm

      Hello Arman,

      It is not possible to file a bug directly to the issue tracker. The correct way is to use a Bug Reporter tool in the Unity Editor. The bug appears on the issue tracker, once we have successfully reproduced it (given it doesn’t contain any sensitive information)

      1. Thank you, Elena.

  4. Thanks Igor, I suggest making a video tutorial about hot to debug a crash and how to report it and include it into the beginners learning sections. In this way, you will take out same not Unity bugs that are trigger by the users or other assets from asset store. For example taking out all the code that do not trigger the bug up to the point, there is one or 2 components that conflict and trigger the bug. Other example Open an empty scene and look how many frame spikes Unity trigger. In this way, the user can report a crash with confidence.

    1. Thanks Alan!

      I’ve passed your proposal to the tutorials team so they can react accordingly.