A look inside: Bug Bash - Find them all!

For a long time, we had Exploratory Testing weeks (ET weeks) as one of the major test phases on every Unity release. The whole QA group would split up into pairs and run the prepared testing tours on the riskiest areas. This approach is great and served us well. After our testers became embedded into the development teams, the focus was shifted to early testing. And the shorter our release cycles got, the less effective we found the ET weeks to be. Becoming an integral part of dev teams meant the whole week spent on a major test phase was a bottleneck. It made planning difficult and slowed the teams down.

Despite all the benefits, it got more and more complicated for the embedded testers to balance the work on the team agenda and block out the whole week for a test phase.

Our main goal was to minimize the stress for the team testers and to avoid the situation where the team perceived a major test phase as an obstacle rather than a benefit. Having all that in mind, we decided to try something new - Bug Bash. Well, not entirely new.... We agreed to have it as a one-day activity. This removed the pressure on teams from testers being unavailable for longer periods of time - but it allowed enough time to immerse yourself in testing the product. This was supposed to be a "test-to-fail" phase and the main intent was to find as many bugs as possible, though the higher severity and high user pain bugs clearly outweighed smaller cosmetic issues.

As an experiment, it was chosen to let testers loose and give them all the freedom in how and what to test. No limitations on techniques, approaches, areas. The blissful paradise for every QA. The only restriction remaining intact is to check for duplicates.

Apparently, one Bug Bash wouldn't be enough to substitute the ET week. Therefore, 2-3 Bug Bashes were planned throughout the release cycle, starting from early alpha and onto beta.

Interestingly enough, the statistics show that the number of bugs found for one day of Bug Bash is roughly the same found during the ET weeks with approximately the same percentage of valid defects filed. The trend has also shown the Bug Bash later in the release cycle finds fewer bugs.

The only downside is that there is no way to get the coverage overview apart from the statistics on the number of bugs per area submitted.

Obviously, testing such a product at Unity is a demanding task. And Bug Bash on its own cannot stand up to this challenge. It is only one tiny part of our efforts in the quest of making Unity even better.