Search Unity

Прошло полтора года после предыдущего разбора инцидентов. Нам нужны были актуальные данные, поэтому мы делаем это снова.

Уточним терминологию: инцидентом считается любое сообщение об ошибке, присланное пользователем через Bug Reporter. Инцидент считается ошибкой после того, как тестерам удалось его воспроизвести. В ходе разбора мы исследуем случайную выборку инцидентов, полученных в течение определенного промежутка времени.

В данном случае рассматривались инциденты, полученные в течение 28 дней после выхода Unity 5.0.1f1. Целью работы было найти ошибки, связанные с выходом этого обновления. Из 2058 инцидентов были отобраны случайным образом 25%.

«Полное расследование» предполагает попытку воспроизведения каждого инцидента, запрос дополнительных данных у приславшего информацию пользователя в случае неудачи, дальнейшие попытки воспроизведения сбойной ситуации с учетом полученной информации. Это требует значительных временных затрат, но в результате мы получаем очень важные данные.

Мы расследовали 491 инцидент и подтвердили 73 ошибки:

image02

Удалось подтвердить только 14,86% инцидента из выборки. Хотя эта цифра значительно превосходит результат предыдущего разбора — 6,83% — этого все же недостаточно для эффективного решения задачи. Так как 85% инцидентов не являются ошибками или дублируют уже известные случаи, мы не расследуем все инциденты подряд вне разборов.

Мы выстроили систему рейтингов для сортировки инцидентов. Рейтинг присваивается с учетом того, насколько подробно предоставленное описание, присоединены ли проекты, скриншоты и другие файлы. Низший рейтинг — 0 (сообщение об аварийном завершении работы с бессодержательным описанием или без него), высший — 5 (подробное описание, проект присоединен). Кроме того, инциденты, переданные корпоративными клиентами, заключившими с нами отдельный договор о технической поддержке, подлежат рассмотрению вне очереди.

Результаты разбора-2015 подтверждают правильность данного подхода.

Как видно из диаграммы, инцидентов с рейтингами 4 и 5 в выборке меньше всего.

image00

Выборка содержит 55 инцидентов с рейтингом 5 и 35 с рейтингом 4. Единственный инцидент без рейтинга попал в выборку по ошибке.

Теперь посмотрим на распределение 73 подтвержденных ошибок по рейтингу инцидентов:

image01

Как видно, большинство подтвержденных ошибок были инцидентами с рейтингом 5. При этом воспроизведенные рейтинги 0 — скорее всего, результат деятельности автоматического анализатора аварийных завершений работы, о котором мы напишем позднее.

Рейтинг   Шанс подтверждения
0 1%
1 0%
2 16%
3 14%
4 20%
5 62%

Рейтинг 4 отличается от рейтинга 5 тем, что инцидент, несмотря на подробное описание, не имеет присоединенного проекта для воспроизведения ошибки. Как видно из таблицы, наличие присоединенного проекта увеличивает вероятность подтверждения ошибки в три раза. Именно поэтому мы присваиваем таким инцидентам высший приоритет. Время на проведение исследований ограничено, и нам необходимо сконцентрировать внимание на тех инцидентах, вероятность успешного расследования которых наиболее велика. На сегодняшний день поступило уже 4510 инцидентов для Unity 5.0.1f1, из которых лишь 10,7% имели рейтинг 5.

Мы продолжаем работать над улучшением процесса обработки инцидентов. Обновление Unity 5.3 будет включать в себя функцию автоматического поиска в Bug Reporter, которая будет предлагать пользователю возможные пути решения проблемы во время заполнения отчетной формы. Мы надеемся, что это нововведение значительно ускорит решение проблем пользователей и снизит нагрузку на нашу команду.

Кроме того, мы собираемся улучшить связь между компонентом Crash Reporter и нашим инструментарием анализа аварийных завершений. В случае аварийного завершения работы пользователь будет автоматически уведомлен о том, известна ли уже возникшая у него проблема и доступен ли исправляющий ее патч, либо о том, что проблема ранее не встречалась и разработчику требуется дополнительная информация о ней.

Мы продолжаем работу над системой обработки ошибок, ждите новостей!

Комментарии закрыты.

  1. Theme list option in Administration panel sholud use Stylesheet instead of Template.~ Line 223echo .$theme[«Name»]. ;This is because some themes have templates (parents) and therefore child themes can’t be selected on the admin panel. Of course other references to the same value sholud also be changed.

  2. That’s a great article, It’s always reassuring when things are breakdown with graphs and such ;)
    Thanks!

  3. Great post. Have you also thought about rating bug reporters? I know systems like this from release engineering when merging contributions. Users could start with 3 of 5 stars. If they report incidents that turn into bugs they increase their «trust» score otherwise it decreases. A 5 star user could be trusted to report important information even if the report is only rated 2 or 3.

  4. vitor alexandre colomby

    Август 22, 2015 в 12:25 дп

    Contract Wars

    1. vitor alexandre colomby

      Август 22, 2015 в 12:27 дп

      battlefield 5

  5. Great insights into your QA internals, thanks.

    We have a very large project codebase here. Although I try to simplify and isolate bugs before reporting, sometimes its just not so easy. We got a couple of crashes and/or problems that are perfectly reproducible with our large codebase (~20 gigs) but it is totally out of the question to upload a copy of that project every time we submit a bug report.

    We already sent access to a git snapshot clone to Rene Damm some weeks ago. If you sent us some pub-key, we can also arrange access to the latest git repo. Is it possible for Unity bug reports to refer to a project by «checkout hash #abc123 from our GIT xyz that we gave you some time ago» instead of attaching the whole project every time? As I understand the posting, this would not qualify for a «Level 5 bug report». (We could also do it the other way round and check-in into some branch of a repo hosted at Unity — if thats easier for you.)

    Also, we often report bugs with our working version of unity — usually a couple of months old. I always check them some days later whether they are still present in the latest Beta (so we might have a reason to upgrade ^^). I usually add some line «Still present in X.Y.ZbW» to the bug report. As I read this blog, this won’t raise the quality level of said bug report, right? Should there be some structured way of updating the version in which the bug still appears?

    1. Hi Immanuel,

      It is OK to send bug reports where you are referring to a project you have already shared with someone at Unity (though I suggest you also mention the name of the person you shared the project with, so that whoever is looking at your bug report knows who to contact to get the project details).

      You are right in that by doing this the rating of the bug is going to be lower and that by adding more details to the bug later on will not increase the rating (something for us to consider). However, making updates to your reports is still welcomed since when someone will get to look at it, there will be as much information in there as possible.

      Thomas already described some of the improvements we are adding to the bug reporting tool and rest assured that we will continue improving on our handling of all incoming reports for the future as well (keep an eye on this space for when we announce more details about our crash analysis tool, for example, that will be pretty awesome)

  6. One thing I’ve found useful is, when you find a bug, to create a new empty project and attempt to reproduce it in the most minimal way possible. The reason for this is twofold:

    1.) You may, in fact, find that you did something wrong and it is not a bug at all. This happens to me a lot.
    or
    2.) Once you have a small repro case, you can attach the whole project to the bug report and it will significantly reduce both the time and bandwidth it takes to send the report, and gives the Unity team a nice small package without all the cruft of your whole game project.

  7. Nice post.

    Unfortunately (for our team), the response from QA regarding most of the bugs i have submitted lately are pretty poor (i believe i provide enough information in the description, as always).

    Most of the issues i raised did not receive any response. I recall things to be different a while ago. Perhaps it has to do with the current release phases and what not.

  8. It would be really nice to get an explanation when issue is marked as Won’t Fix.
    e.g. http://issuetracker.unity3d.com/issues/android-several-unnecessary-permissions-are-added-to-app-manifest

    1. I agree. The description should have been updated with this information.

  9. Hi Thomas, and thank you for shedding some light on a topic that interested me for a while, how it works.

    I am curious, what do you do about bugs like these?

    Game Center achievement not working: http://forum.unity3d.com/threads/problem-with-game-center-achievements.310817/

    I have reported it a few times, there is a (now mature) forum thread.

    What is some other way to bring this to QA’s attention (or whomever it concerns)?

    Thank you!

    1. I think you did the right thing by posting on the forums. Helped out a lot of people.

      The incident in the thread had not been looked at. Why that is the case, I can’t say, but it was rated 4. There’s a good description on it, but no project attached, which is why it is going one down.

      Anyways, it looks interesting and we’re going to look into it.

  10. I wrote a post last month about how I think the bug tracker could be improved:
    http://forum.unity3d.com/threads/improved-bug-tracker.339570/

    The automated search and crash analysis lookup sound good, hopefully you’ll be adding some interaction possibilities too.

    1. The post has a lot of good suggestions, of which many are along the same lines we are thinking. It will take a while, though, because we have to integrate our backend systems with the login in the editor and then we can start playing with the data.

      But this takes time. Longer than you would expect, because this train can’t stop for just a single day, so we must be extremely careful.

  11. A few years ago I sent a bug report for a problem that was only reproducible on an Intel GFX chip that Unity had already said would be unsupported in the next release (3 — 3.5 — 4, I can’t remember which it was) and they not only reproduced the bug but fixed it in time for the text minor release less than 7 days later.

    From that moment on as soon as I saw a bug I thought could be in the engine or editor I’d spend 15 mins trying to create the smallest project to show it in effect to submit a bug report on.

    By acting fast and taking the time to fix a bug that was only ever going to effect a small portion of their developers and their final customers they turned me into an unofficial part of the QA team :)

    Those bug reports really to get acted upon people.

  12. I always try my best for bug reports, to help the recipient save time, but also to make sure I know how to reproduce the problem exactly with an example as solid proof. By doing that, I also make sure I’m not just misusing the software, sometimes I learn that it was my mistake, just by writing the description. Time and effort saved for everyone!

    The only bug report I filled for Unity was a 5 I guess, from what you say in this post, and I got an answer only 24h later. It said the bug is real but low priority.

    1. … forgot to add: and Unity staff also provided an easy workaround, so good job, and thank you guys!

  13. Very interesting.

    I wonder if you can give a bit more information to get a 5 point rating to the bug report (when is the description good and when is it enough to submit a small test project).

    I’m asking because I’ve got some tickets (with descriptions and test projects), but I get hardly feedback to them (mostly being in a monologue). Don’t understand me wrong: I think the QA process has improved a lot from you guys in the last 1-2 years, but I find it hard to understand when a ticket is helping you to find the bug and when not.

    I want to produce quality bug tickets, so you have an easier time to find the problem and solve it (which would make me happy, ’cause they are affecting me and my project). If I create a bug report and invest time into it, it’s a bit frustrating to see some of my tickets going into the void (or at least it seems to me).

    So, transparceny is a good thing! :)

    1. First of all, thanks for submitting the bugs, Pahe.

      If you have provided a description and a attached a project to your bug report, that would get it high on our priority.

      What I didn’t mention was that we also take the version number into consideration, so older versions get less and less attention. And when I mean «older», it can be only a few months old, because that means the codebase has moved SO much in the meantime that we can be chasing a bug which is no longer valid. To give this some context, the Unity codebase gets more than 4500 changesets committed every month, so when the latest official release is 5.1.2f1, that codebase is over 9000 (yes, I said it) changesets behind the one we are currently working on in 5.2. So, submitting reports on the latest possible version is another good practice in order to get the incident prioritized.

      1. Is this per major version? I.e. the latest 5.x release and the latest 4.x release? I understand that 5.x gets the most love, but I’m wondering where 4.x fits in, considering some of us are stuck in 4.x for a while (in our case to support iOS 5.1.1)

        1. 4.x only gets attention if we have issues from our enterprise customers OR there is an issue with IL2CPP. In some cases we also back port a fix all the way to 4.6, but it is getting very rare. 4.x will sunset completely by the end of this year.

      2. So lets say I submitted a bug for 5.1.2 (with repro and description) and it didn’t get a response. And the bug still exists in 5.2. What is a good way to make sure a bug that’s been around for a long time still gets attention?
        Submitting another duplicate bug with 5.2 doesn’t seem productive for anyone involved.

        1. You’re right, it doesn’t. We will focus on 5.1.2, but if you were to submit a bug for 5.1.0f1, it has a very high risk of not getting attention, especially if it is also rated below 5.

          As you can tell, there are a ton of different priorities at work, so the best pointer I can give is to be on the latest version. Patch versions if they apply to you, but otherwise latest version updated through the editor.