A Look Inside: Asset Store Bash, upgrade testing using Asset Store packages
Asset Store Bash is one of the major test phases performed as part of every major Unity release.
It has two main purposes:
- Testing how the new version of Unity handles upgrading of significant variety of different assets and projects
- To ensure that our recent features and changes haven’t broken the most popular packages created by you, our dear customers
We extensively test all new functionality added to Unity and have a number of various test activities to make sure old functionality is kept intact. Update testing is focused on specifically figuring out if older projects will run nicely in latest version and if the upgrade process itself is as effortless as possible. Sometimes adding new features requires changing things or deprecating older functionality, but we are always focused on making it as easy and painless for you as possible.
For upgrade testing at Unity we have a number of different approaches from suites of automated tests for script updating and backwards compatibility, to some big user projects being manually upgraded to newest version. Asset Store Bash fills out the hole in the middle, with test data being different packages found on Asset store, so it has a lot of variation with low complexity.
We started doing this as an experiment when stabilizing 5.3 in the 5.3.4 release, and it has become normal practice since then. At first, the list of packages selected was simply based on their popularity, so mainly the most downloaded and most purchased assets were tested as they affected most of our client base.
Amount of packages tested has been increasing with every version, with the quality of selection also improving.
From 5.5 we started hand-picking the packages to cover all the main features and areas of the product. On average there is 10 packages per area, with 20 or 30 for bigger areas with more embedded testers. Testers gain complete ownership of the assets selected for their features. They gain freedom to select the packages they consider most suitable for testing of their areas, and mark unsuitable packages for the next release.
This has dramatically increased the suitability and overall coverage of the tests. Instead of having a few managers selecting packages in bulk based on their popularity, people responsible for the areas have to hand pick only the 10 packages most suitable for their features.
Testers note down any issues with the assets and report the bugs. This includes both Unity bugs and problems with the package itself. We usually try to drop the publisher an email with upgrade tips if their package was affected by some new feature or improvement.
Furthermore, tester will mark the status for current version (5.6), compare it to previous one (5.5), and mark if they find the package useful for testing any particular areas.
Realizing that not all of popular packages will be selected, but are highly valuable as packages getting that much popularity are usually highly important for you with a lot of people downloading them every day, we’ve decided to always test the top 50 free and top 50 paid packages.
That wraps our selection and performance of the Asset Store bash. If you have any questions or ideas – feel free to express them in comment section below.