Search Unity

A Look Inside: Asset Store Bash, upgrade testing using Asset Store packages

, July 3, 2017

Asset Store Bash is one of the major test phases performed as part of every major Unity release.

It has two main purposes:

  • Testing how the new version of Unity handles upgrading of significant variety of different assets and projects
  • To ensure that our recent features and changes haven’t broken the most popular packages created by you, our dear customers

We extensively test all new functionality added to Unity and have a number of various test activities to make sure old functionality is kept intact. Update testing is focused on specifically figuring out if older projects will run nicely in latest version and if the upgrade process itself is as effortless as possible. Sometimes adding new features requires changing things or deprecating older functionality, but we are always focused on making it as easy and painless for you as possible.

For upgrade testing at Unity we have a number of different approaches from suites of automated tests for script updating and backwards compatibility, to some big user projects being manually upgraded to newest version. Asset Store Bash fills out the hole in the middle, with test data being different packages found on Asset store, so it has a lot of variation with low complexity.

We started doing this as an experiment when stabilizing 5.3 in the 5.3.4 release, and it has become normal practice since then. At first, the list of packages selected was simply based on their popularity, so mainly the most downloaded and most purchased assets were tested as they affected most of our client base.

Amount of packages tested has been increasing with every version, with the quality of selection also improving.

From 5.5 we started hand-picking the packages to cover all the main features and areas of the product. On average there is 10 packages per area, with 20 or 30 for bigger areas with more embedded testers. Testers gain complete ownership of the assets selected for their features. They gain freedom to select the packages they consider most suitable for testing of their areas, and mark unsuitable packages for the next release.

This has dramatically increased the suitability and overall coverage of the tests. Instead of having a few managers selecting packages in bulk based on their popularity, people responsible for the areas have to hand pick only the 10 packages most suitable for their features.

Testers note down any issues with the assets and report the bugs. This includes both Unity bugs and problems with the package itself. We usually try to drop the publisher an email with upgrade tips if their package was affected by some new feature or improvement.

Furthermore, tester will mark the status for current version (5.6), compare it to previous one (5.5), and mark if they find the package useful for testing any particular areas.

Realizing that not all of popular packages will be selected, but are highly valuable as packages getting that much popularity are usually highly important for you with a lot of people downloading them every day, we’ve decided to always test the top 50 free and top 50 paid packages.

That wraps our selection and performance of the Asset Store bash. If you have any questions or ideas – feel free to express them in comment section below.


5 replies on “A Look Inside: Asset Store Bash, upgrade testing using Asset Store packages”

Testing editor Metal support using asset store stuff would be good, especially at the stage where metal support in the editor is experimental. I’m thinking specifically of assets that use compute shaders. In many cases getting them to work in metal just requires any standard shaders to have their pragma targets changed to 4.5 instead of 5. Which brings me to another point – it would be good if Unity could reach out to developers of these assets to help them get mac/metal support done, both in cases where the necessary change is trivial and also with ones that seem to reveal other issues with metal compute shader support. I’m thinking of assets such as Fluidity, Cocuy, and TC Particles.

I hope you are adding to the list assets that are VR capable like Beautify, Volumetric Fog & Mist and Liquid Volume. These assets employs advanced shader features like full screen post process and CommandBuffers with Single Pass Stereo support. If you break anything in 5.6 or 2017 or a future Unity release that will show up in these assets.

I’ve always wondered if this was done.I’m glad to know that the unity team is proactive with the assets on the UAS, one of the most valuable components of unity to me.
Good job

Comments are closed.