Search Unity

Continuing the Release Management Series, let’s talk about how we changed our bug prioritization system for 5.4 and why.

In recent times, we have been focusing a lot on quality. One common, though inaccurate, measurement of quality is the bug count. In the last two years, we’ve been crushing out over 11,000 bugs per year (!), but even so, we never came close to reaching zero bugs. We don’t make shrink wrapped software running on fixed hardware platforms with specifications that can be locked in. Aside from cases of our own optimizations or fixes creating new bugs, new issues arise daily. There are countless cases, some of them not even coming from our own code base.

So, the question we ask ourselves is, “Which bug do you fix first?” Unity has so many dimensions to work with, so is it fair to compare a “small platform bug” against a “editor usability bug”? Under the old system, we would track a priority, but the teams defined the priorities individually, so it was decidedly inconsistent. What is a shipstopper for one group does not make it a shipstopper for all. While some teams with larger overviews and ability to cross spaces had a consistent priority system, most did not and what was often organizational team priority was assigned at a global scope. You might see how this comes to be a release management problem.

So, in comes “user pain”. The “user pain” idea can be found in a lot of places, but the seeds for us started with another Director of Development in R&D, Erik Juhl, recalling a methodology used during his MMO days where the team would assess the risk of fixing a production bug. It went something like this:

Risk Assessment = Tech Risk + QA Risk + Business Concern + Community Impact


In that situation there was a threshold to reach in order for the whole group to believe a bug was worth fixing immediately. This worked for its purpose of “live” issues in a service and case by case analysis, but it doesn’t exactly scale to bug databases. However, the idea was something we latched on to.

Finding A Formula

With this example in mind, our release management sampled all the old priority system “Priority 1 (Drop everything now)” and “Priority 2 (Shipstopper)” bugs marked for fixing in 5.4, which amounted to some ~150 bugs at the time in mid-development.

We started with something like this:

User Pain = (5 – Severity) * IsRegression + QA Risk + Technical Risk

Here, Severity and IsRegression were information that previously existed in our bug tracking system (see below about Severity changes). “QA Risk” was an attempt to describe the risk in the ability to test the bug. “Technical Risk” was attempting to capture the complexity and “halo effect” (the likelihood the fix will influence another part of the system).

In looking at the sampling of bugs and calculated pain, the list ordering looked wrong from our perspective. Mainly, we decided to drop QA Risk & Technical Risk as they are things only for release assessment, “if we should accept a bug fix”, instead of user pain, which should represent “should I prioritize fixing this.”

So, we next refined the formula to something like:

User Pain = ((5 – severity) * (Platform Importance * Prevalence) ) * (1 + IsEnterpriseSupport + IsUpcomingPlatform) * (1+IsRegression) + IsInstability * 100 + Votes

 

We introduce a number of things here:

  • Platform Importance – to separate individual edge platforms spanning to cross platform
1 = default Single platforms, i.e. Tizen, PS4, etc.
2 = Core Platforms Desktops + iOS + Android + WebGL
3 = Editor Greater than Core Platforms, because working on making the game trumps shipping a particular platform
4 = Cross-Platform Most or all platforms
  • Prevalence to consider how widespread it is
1 = Affects few or very select users
2 = Affects some, but not majority of users
3 = Affects a majority
4 = Affects everyone
  • IsEnterpriseSupport – addressing Enterprise Support requests and their importance
  • IsUpcomingPlatform – working with various partners, this is a way to highlight issues that might need to be tracked
  • IsInstability – is a bug causing internal development slowdowns by introducing an automated test instability. While not obviously user pain, it’s indirect and very very impactful which is why it gets such a high rating. These can cause significant slowdowns that impact how much we can fix or improve quickly.
  • Vote – a simple vote counter collection from the Issue Tracker

This formula gave a nice spread of bugs that seemed to properly give weight to appropriate bugs, and we worked with this idea for a while. Over time we made the following adjustments:

  • IsUpcomingPlatform was dropped, as it would carry excessive weight beyond the life of the initial release. Plus it was expected that the team in charge of the platform would be owning, raising concern and chasing these bugs.
  • IsEnterpriseSupport was dropped. This was in recognition that the Enterprise Support team in conjunction with our Sustained Engineering were already on top of these marked bugs. We were running into each other looking at the same bugs with extra engineers now looking at bugs that someone was already working on.
  • “Release Manager” tag was added to give weight to items like Known Issues or other items showing up lower in the ranks that were of high importance to users, but scoring did not reflect so.

So now we’ve arrived at:

User Pain = ((5 – severity) * (Platform Importance * Prevalence) ) * (1+IsRegression) * (1 + RelMgr) + IsInstability * 100 + Votes

We started with a threshold of 54, on the basis of:

  • Severity (Major functionality impacted) >= 2
  • Platform Importance (at least Editor) >= 3
  • User Prevalence (Majority) >= 3
  • Is Regression (True) = 1

However once we went on and removed the Enterprise weighting, we dropped the threshold to 48 by arbitrary inspection. This is the current state of things, but we recognize it’s a living idea and may change over time.

The one other adjustment was made that our Severity classification was somewhat broken. It would allow minor issues to trump functional issues on the basis of “no workaround available.” Here’s the shift of mapping we decided to implement along side of things:

Severity Old New
1 Crash or major loss of functionality Crash, freeze, data loss or work stops
2 No workaround Core functionality majorly impacted
3 Workaround is possible Secondary functionality broken
4 Minor or cosmetic issue Minor or cosmetic issue

Adoption

Across R&D there’s been appreciation of the new approach. Many teams have adopted it and are quite pleased thus far. QA members and release management have gone through lots of the existing bugs to assign appropriate values for the newer variables we had to add.

With the adoption of this process, people are starting to step away from the old priority mechanisms and view the bugs in light of how they impact the users.

image00

What’s interesting about the graph above, is that painful bugs still counted for less than our per team assessments of priorities. The “Pain” line grew because we expanded scope of bugs being assessed far beyond 5.4 P1 + P2. It eventually encompassed any bug listed from the past 5.0 to the future 5.5 (5.x) for all old prioritizations of P1 – P3, plus our bug triage queue. The numbers still were less than the “high priority bugs” of our past weighting. Part of the up and down was the incorporation of 5.3 items and then dropping of the IsEnterpriseSupport bugs which already had coverage.

Thus far, what we’re tracking with user pain plus a couple other items like upgrade issues is showing to be largely the main user concerns.The current feeling about the release is fairly positive without correlating as strongly to the number of P1s and P2s.

Future

So, what’s next for the “user pain” system?

Well, currently it’s a fairly manual analysis for us, and not fully integrated into our bug system. Also, we need to take a pass at all the older bugs that haven’t been assessed for platform importance and prevalence.

Additionally, we still know there’s tweaking to the formula to consider. We may have missed some aspects to include or are still giving a lot of weight to area we shouldn’t. Also, there’s a great benefit to being able to change the formulas when needed, so it better reflects the issues of the moment. We know that the pain index is not an absolute measurement, but guidance to help show how we might consider prioritization.

Lastly, we’ve started experimenting with the idea across other areas including the release risks for acceptance of fixes during RC/patch. We’re also thinking how this might be useful to prioritize features or re-factors.

It’s all still an experiment and thus far current results and perception is that a more unified system of prioritizing is helping. We’ll continue the experiment forward and update you some time in the future on how things keep going.

For the next round, we’ll talk about some internal tooling changes.

18 Comments

Subscribe to comments

Comments are closed.

  1. I think that one of the problems is that you Guys don’t develop games on your own engine so you can’t actually tell what is worth fixing and what is not. I know that you have game demos and tutorials but this does not equal a published game that went through full development pipeline. I’d like to see Unity Games by Unity3D which would provide feedback for the engine dev team. It works really well for in-house game engines and companies like Unreal and Crytek. Cheers!

  2. that’s true! UnityBug3D ! has reduced lots its’ Programmer’s lifetime .
    is it possible charge unity3d in law ? for so many bug kill Programmer slowly

    Today my boss ask me to learn UE4. it is really piss me off. hey! Unity! when will make your light and shader better than UE4 ,I am count on it !

  3. I’m still skeptical. It’s not the first time Unity tries to tackle bugs and fails miserably, there are years old bugs that have not been addressed. If this changes how Unity prioritizes their development to include more fixing and polish them I’m all for it. But while I like the idea that this re-prioritizes the fixing to be more “user-aware” they have failed far too many times before.

    I’ll be believe when I see it.

  4. Now you need to open up the voting. Having a limited number of votes doesn’t accomplish anything useful. Being able to dump multiple votes into an issue is also a needless complication. Let users apply a single vote to any issue. That will give you a much more accurate read on what is important.

    I have all my votes dumped into a couple of items that I just hope will get attention some day, and it’s extremely frustrating to run across a new issue and not be able to raise my hand and indicate there’s at least one more project out there impacted by the problem.

    Opening up voting won’t dilute the value of votes either. If somebody sits down and votes on every one of the top 1000 items, there is net zero effect. A rising tide lifts all boats, etc.

    Votes aren’t a scarce resource and artificial scarcity just reduces the value of voting, probably to the point that most people don’t even bother.

  5. You should include a time parameter to prevent bugs “starvation”

  6. Thanks to this bug, I spent x10 time aligning all the objects in all the levels of my game while thinking that I was using prehistoric software (having to zoom all the way in the scene in order the vertex spanning to work fine, and then zooming all the way out, doing that for every new object I wanted to position in the scene). https://issuetracker.unity3d.com/issues/vertexsnapping-different-levels-of-accuracy-at-different-zoom-levels (If this would’ve worked as expected, I would’ve saved a few days work) Btw… it’s going to be one year and the bug isn’t fixed yet. :(

  7. Would really like to see a editor dev category in prevalence – in place of 4 or at least 3.
    Because what we editor devs are able to make with the abilities you give us does in the end affect almost everyone. And there are still some ceavats that are really, really annoying and sometimes double developement time for us editor devs, trying to develop workarounds…

  8. Now that you have [User Pain] you want to evaluate [Effort] as well.
    Effort is the amount of man-hours that will be spent on reproducing, fixing, testing the bug (including the risk of introducing secondary bugs)
    Total effort is limited (available resources) so you want to select bugs that the sum of User Pain reduced as large as possible.
    This will be some sort of [User Pain]/[Effort] metric.

    tl;dr if you can fix 10 cosmetic bugs in the same time as one serious bug then it sometimes worth it.

  9. I love how you guys give info, about how you actually are running things. Few other companies are open about it. A lot of companies hide this information behind closed doors + NDA’s.

    I really love these kind of posts! :) Please keep making them.

  10. Still waiting for a 3 month old bug that’s been fixed in 5.3 but yet to make it in to 5.4, after 3 e-mails and no replies.

    So I’m all for much needed changes to this bug tracking system.

  11. LOL. yes. and is not only a crash. Is not only pain; is n° user * time * pain)square since taking out time from making other. Imagine the ripple effects of 11,000 bugs per year + bugs that for Unity are not consider as bugs but users thinks are bugs. One example is Y up and no way to change it in Unity settings. A bug is a lack an omission. And each developer producing 1000 lacks x Yr. Is it because are running too fast? If you go fast in a light car and then you stop for a coffee, probably that slow clever old truck driver will pass you.

  12. “some stuff + Votes”

    So an issue with 5000 votes trumps everything and will be on-top of your list all the time?

    1. I’m sure they would re-address that section in the future if they were getting many issues with 5000 votes. Right now the most voted issue has 241 votes (which is fixed) so it’s probably a good measurement for them right now.

    2. yeah I can see how this could be a problem. Highly-voted issues in Unity Feedback tend to be stuff like “We want a Make MMORPG button”

  13. Such a system of ‘user pain’ will only lead to exaggerated claims of disasters by users looking to jump the queue and not solve the problem at hand.

    A better approach would be to improve the performance metrics gathered by the Unity Editor and concisely send them with the error report.

    Also, most of the time I have a bug it’s due to running out of memory or a namespace collision or an absence of a needed library or a dll inclusion problem – all those things happen frequently if a use imports a the wrong combination of Unity asset packages into their project – so I’m doubting the validity of many of those filed bug reports or at least the conjectured cause of the bug.

  14. Robert Cummings

    August 17, 2016 at 4:23 pm

    5 = it crashes
    4 = it runs like shit
    3 = usability
    2 = feature
    1 = polish
    https://www.youtube.com/watch?v=tt5QEynwjKg

    1. new content:
      5 = it crashes (bug report and Unity Internal Discussion)
      4 = it runs like shit (bug report and Unity Beta Forum)
      3 = usability (bug report and Documentation)
      2 = feature (bug report and Video explaining the new feature )
      1 = polish (bug report and Video Tutorial or Live Training about it and how to use it!)
      0 = clean if (bug report = 0)

      I put maya as example: Maya has old boolean bugs since 90´Alias and will be there for ever at this point; That makes a software old. Is better to fixing old bugs before making more. Makes the software stable and ready before the author goes or moves on. Only experts users can find in the stage 1 problems. but including that problems must be fix till there is 0 bug reports. In this way expert users can make much more. I can´t use Maya booleana because I can´t push them up to my limite. The limitation in this particular example is the bug. But usually the software developer how make the new feature thinks is not important to has 0 bugs. Thinks is more important to have new content.