Search Unity

For a long time, we had Exploratory Testing weeks (ET weeks) as one of the major test phases on every Unity release. The whole QA group would split up into pairs and run the prepared testing tours on the riskiest areas. This approach is great and served us well. After our testers became embedded into the development teams, the focus was shifted to early testing. And the shorter our release cycles got, the less effective we found the ET weeks to be. Becoming an integral part of dev teams meant the whole week spent on a major test phase was a bottleneck. It made planning difficult and slowed the teams down.

Despite all the benefits, it got more and more complicated for the embedded testers to balance the work on the team agenda and block out the whole week for a test phase.

Our main goal was to minimize the stress for the team testers and to avoid the situation where the team perceived a major test phase as an obstacle rather than a benefit. Having all that in mind, we decided to try something new – Bug Bash. Well, not entirely new…. We agreed to have it as a one-day activity. This removed the pressure on teams from testers being unavailable for longer periods of time – but it allowed enough time to immerse yourself in testing the product. This was supposed to be a “test-to-fail” phase and the main intent was to find as many bugs as possible, though the higher severity and high user pain bugs clearly outweighed smaller cosmetic issues.

As an experiment, it was chosen to let testers loose and give them all the freedom in how and what to test. No limitations on techniques, approaches, areas. The blissful paradise for every QA. The only restriction remaining intact is to check for duplicates.

Apparently, one Bug Bash wouldn’t be enough to substitute the ET week. Therefore, 2-3 Bug Bashes were planned throughout the release cycle, starting from early alpha and onto beta.

Interestingly enough, the statistics show that the number of bugs found for one day of Bug Bash is roughly the same found during the ET weeks with approximately the same percentage of valid defects filed. The trend has also shown the Bug Bash later in the release cycle finds fewer bugs.

The only downside is that there is no way to get the coverage overview apart from the statistics on the number of bugs per area submitted.

Obviously, testing such a product at Unity is a demanding task. And Bug Bash on its own cannot stand up to this challenge. It is only one tiny part of our efforts in the quest of making Unity even better.

Comments are closed.

  1. It is the end of the Unity 5 lifecycle and 5.6 looks like the original Unity 5 pre-release. Everything is marked with preview, beta, or experimental to rally interest in Unity 2017. It sounds like you have already subjected customers to being a part of testing by releasing trial builds in place of release versions.

    The few issues I have attempted to submit in the past were fixed within hours, but only after the month it took the support staff to understand them. I should not have to resort to drawing pictures and making a PowerPoint presentation because simple phrases like “the file is listed, but it is not included” are incomprehensible. That said, this appears to be an entire day reproducing an error that is likely just caught up in someone else’s “needs further details” queue. It would be time better spent resolving known issues.

    The number of bugs decreasing throughout the lifecycle is not something you should have to advertise, as it should be implied. Longer testing periods are less effective within shorter release cycles, but only because less focus is placed on identifying underlying problems. The issue is that rather than shortening the release cycle for publishing resolved issues, Unity has attempted to expand the Editor and simply keep the bugs maintained. This leaves the number of underlying bugs growing exponentially and explains why a day of “Bug Bash” testing would find nearly as much as an entire week of ET. More issues are being overlooked because they did not create “obvious” problems.

    Perhaps Unity should stop setting deadlines before having a grasp on the timeline. The release times keep being accelerated, but the progress keeps falling behind. If it takes a day, then a day is all you need. If you find there is too much for a day, then people will appreciate taking a second day over releasing garbage.

  2. Unity has the worst QA team. It just shows how poor the editor code base is that every time something new gets added, it can be broken in multiple ways.
    Take for example when they added the game view zoom. Its such a simple feature. But wait, even with such a simple feature, there is unusual bugs – The game view gets zoomed 1.5x for no reason at random times. Its like an amateur writing code for the editor.

    1. Sick of the negativity on every unity blog post.

      I use Unity and love it. I’m understanding when things break.

      Be a nicer person. Use something else if you don’t like it. You don’t need to spread your contagious negative attitude on blog posts for products you dislike.

  3. hablo español no todo puede ser ingles l speak spanish, not everything can be english

  4. Robert Cummings

    March 24, 2017 at 10:02 am

    Seems strange so many bugs keep cropping up. You seem to be fighting bugs more than actually developing though. But maybe that makes sense.

  5. Vladislav Ivanchuk

    March 24, 2017 at 6:13 am

    Hi dear unity team! I don’t understand one thin!

    We really wait 5.6 unity! But now we can’t test our game on target devices – iOS & Android. Why ? Because 5.6 beta have bug, witch crush player

    Known Issues
    > Particles: Game crashes occur during Play mode when a Particle System is a sub-emitter of more than one other Particle System. (857065)

    https://issuetracker.unity3d.com/issues/crash-when-a-particle-system-is-sub-emitter-of-more-than-one-other-particle-system

    How it possible, what you not fix bug over 4 moths (may be more) with crush player???

    I think as result – when 5.6 will be released, it will be have tons of bugs!…

    1. That bug is fixed in the next release candidate build.

  6. Love the Bug Bash idea!

    Can you talk about what techniques and approaches were favored most by your teams? Also, between alpha & beta builds do your teams’ techniques & approaches change much?

    1. Hello, Brian.
      Our teams are doing a lot of in-branch early testing even before a feature hits an alpha build. The feature testing goes through some levels: from checking a basic functionality on simple data to an extensive usage with large projects and complex data. Non-functional aspects are also part of that effort. As well, teams make sure to have automated tests in place.
      During the beta phase we run a regression pass, upgrade testing, Scenario week. You can read more about our test activities and processes here