Search Unity

As outlined in the Unity Engine QA Process blogpost, every Unity release goes through a number of test phases before it is released. Here I would like to dive in and present one of them: Scenario Test Week (dubbed FTP v2 in the previous blogpost).

Simply put, it’s testing by creating games. We prepare for it by creating a number of scenarios – a short pitch for a game project. Examples could be “3D Racing Game”, “2D Platformer” and so on. Then we group participants into teams, assign each team a scenario and then the teams have one week to create a game based on that simple premise.

For testing our most recent release (at this time of writing), Unity 5.5, we had 14 scenarios and 53 participants, including testers, student workers and developers.

But scenario testing is more than a game jam. It’s a testing phase with a very specific goal: Find bugs by using the Editor in a credible way. We have made a breakdown of Editor features and platforms and mapped it to the different scenarios to get as wide coverage as possible. We make sure that features are covered by more than one scenario. By having scenarios be different game genres, we ensure that the features are used in different ways.  

Areas mapped to different scenarios/teams for 5.5 Scenario Test Week

Different test approaches find different problems. We have more than 60000 automated tests with millions of executions each month. Our manual regression suite – Full Test Pass – consists of thousands of manual test-cases that provide a comprehensive sweep of the functionality in Unity. This is all great and invaluable! But broadly speaking, these kinds of tests tend to exercise functionality in isolation. Some errors appear when you combine features. Or they appear over time. Something might work the first five times you do it, only to fail the sixth. Furthermore – again, broadly speaking – test automation and manual test-cases repeat the same test over and over again. And while the scope of what we achieve with automation is impressive, it’s still only a tiny fraction of the theoretical possibility space for how Unity can be interacted with.

With Scenario Testing we address some of these shortcomings. When you create a game, you don’t simply methodically go over each feature of a product. You get repetition of actions, you get randomness and the Editor will be interacted with in ways that are impossible to predict when designing a feature or even a set of tests. And most importantly, the Editor is used in a way that is credible. Even if it’s an approximation, we get to feel some of the joy and frustration that our end-users feel. This is powerful. Chomping on your own dog-food is always a good thing…

Bugs reported during 5.5 Scenario Test Week

Here is an example of the kinds of feedback yielded by scenario testing. One of the scenarios was to create a Real Time Strategy game (screenshot below). It was intended to obtain coverage for AI, Particles and Terrain. However, once the game was taking shape, naturally the idea came up to add multiplayer. And very quickly the team ran into problems adapting their game to the current networking system. But the upside was that they were able to take this feedback to the networking team who could use it to inform future decisions regarding networking in Unity. We got this feedback because the conventions of a particular game genre inspired us to use our game engine in a certain way. And we got it even though networking was not part of the coverage scoped for the scenario. Automation or manual test-cases will not give you this kind of information.

We are continuously looking for ways to improve how we do Scenario Testing. One thing we are looking at is continuing work on some of the scenarios across multiple Scenario Test Weeks (i.e. multiple versions of Unity). Until now, we started over with new game productions each test-phase. We hope that this can give us a better approximation of both upgrading Unity during a production and also working on a project for longer than hacking the first prototype together.

We don’t presume that our little scenarios can accurately replicate the experience of working with Unity on a real project. Still, we feel that we can make a pretty good approximation Scenario Testing. And – not to forget – it’s actually a fun way to test Unity.

Here are a few small samples of projects that were created during Scenario Test Week for Unity 5.5. Note that in addition to the coverage mentioned for each project, all projects also used Unity Collaborate and Unity Cloud Build. We dog-food our services offerings as well. Asset Store is also tremendously helpful as it allows teams to use high-quality assets in their projects. Not only does this boost the production value of the games, it also contributes to testing by allowing more complex and realistic data/assets to be used.

FPS game Team size: 4 people. Target platform: Desktop. Focus areas: Audio, General Graphics, Substances, Post-processing

3D RTS Team size: 4 people. Target platform: Desktop. Focus areas: AI, Particles, Terrain/SpeedTree integration

 

2D JRPG Team size: 3 people. Target platform: Android. Focus areas: 2D, UI, Particles

 

2D Platformer Team size: 3 people. Target platforms: iOS, TVOS, PS Vita. Focus areas: 2D, Animation 2D, Physics 2D

 

VR Museum Team size: 2 people. Target platform: VR. Focus areas: Graphics – general, Lighting, Post Processing

 

Slender Man clone Team size: 3 people. Target platforms: PS 4, VR. Focus areas: Graphics – general, Terrain/SpeedTree integration, Substance integration

 

22 Comments

Subscribe to comments

Comments are closed.

  1. Unity 5.4.4f1 crashing after splash screen still is exist in some android devices.

  2. Unity 5.5 – 5.6

    https://forum.unity3d.com/threads/gfx-waitforpresent.211166/page-5

    create a canvas with transparent images on it(panel an …)

  3. In the Sony Z3 game some times crash without any reason ?!
    Also Samsung j1

  4. Could one of your scenarios perhaps be ‘try to use Unity with Perforce as your source control provider’ ?

    1. Try resetting GUI layout and/or deleting the Library folder and reimporting the project.

  5. Nice looking RTS demo but really you need multiple AI players competing and ideally over networked multiplayer.

    I wrote a basic RTS 3x AI bots vs 1 player here -> https://arowx.itch.io/simple-rts

    Also Unity lacks AI out of box e.g. FSM and or an Agent based system.

    You could leverage the UI and logic of the Animation system to make a good basic and extensible AI FSM subsystem.

    Also very high Unit counts battles like Ashes of the Singularity should be tested for performance.

  6. Unity 5.4.4 release date Please ?
    Splash screen in the 5.4.3 (android) in long loading time become black ?

  7. As a programmer that has been embarrassed at very important field sites in front of VIPs by programmers that swore up and down that they tested and re-tested every unit test and the entire regression test only to watch a system boot and know that they didn’t even boot the software to test it I really like your increasingly comprehensive test suites.

  8. Robert Cummings

    January 4, 2017 at 3:39 am

    Hi,

    I also love you’re doing this but you guys stop where the problems typically begin, and don’t push things. Basically Unity tends to fall over when pressured or pushed, not when you do things within margins.

    Basically do something high end then try making it run on low end. Basically, make Adam demo work on a mobile at 30fps solid would be a good start. Next up, try to make a decent sized Unity-mmo which should be graphically very intense – you will probably hit all number of issues trying to optimise that case scenario.

    If I’m not mistaken, all your tests are designed with a target in mind. When I make a game, I target quite a bit further than the target hardware then optimise it to fit, and it’s this optimising to fit that requires creative solutions which breaks things.

    Unity never breaks if you use Unity with it’s default functionality…

  9. I love that you guys do this. But my team looked at this and rolled their eyes. And here is why:

    The places where Unity tends to really fail us is seldom if ever during the prototype phase. It’s when we try and push to polished product that works properly on all the target platforms. What you have here is roughly just testing games at a prototype level, and sorting out the bugs inherent in that.

    Great, but super insufficient to really sus out the real offenders lurking beneath. Does this tell how how a game will work on the literally thousands of different Androids and OS configurations? Or even within the iOS family (iPad 3 anyone?).

    Let me give you an example of one such bug:

    We switched from Prime31 to Unity’s built-in IAP. Great. Worst, maybe 99.9% of the time. But ever once in awhile, it misinterprets the local currency price of an IAP purchase. For example, if something is say 450.312 Rupiahs, it’ll sometimes just not realize that a decimal point (or in the real-world case, the comma) is meaningful and instead report 450321 Rupiahs. Off by… oh a little, yeah?

    Can we recreate it? Nope. It just happens…. very rarely, sporadically, with little pattern (except if it happens to one individual it usually, but not always, keeps happening with him). Something with an interaction with the perhaps how the device interprets the localization settings of the OS (maybe their country is Malaysia, for example, but their Language is English). Not sure, it just happens.

    This is the sort of thing that wrecks our schedules and leads us to madness, but is nearly impossible to make a repo for or even get an visibility into. We’re still trying to find any clues on this one to make a meaningful bug report out of it.

    I don’t know how you’d make testing for this, but I guarantee these little gamejamy efforts won’t turn this sort of thing up. And I hope you don’t take solace in these scenarios once they hit zero bugs that you’re at gold master. You are not.

    1. Here here

    2. “And I hope you don’t take solace in these scenarios once they hit zero bugs that you’re at gold master. You are not.”

      We do not :-)

      Issues of performance, scalability and large teams are very much concerns of ours and we are putting lots of efforts into this.

      But this is not the goal of scenario testing. We do not presume that making one week prototypes will find issues of a real-life large-scale production. As mentioned, we have other efforts with that as the goal. But there is a reason why the early prototype stage is a relatively smooth experience (one might argue this was not always the case). That is in part because of efforts like this, comprehensive automation suites and the other testphases the Engine goes through.

      One thing (among others) that we have pushed for lately is to make sure that new features have been tested in realistic, real-life context. We believe that you will the results this and our other test efforts in the form of more mature new features released in the new year.

    3. Nicholas Richards

      January 4, 2017 at 9:46 pm

      Hello David,

      Thank you for raising this finding; discovering a single sporadically incorrect currency magnitude for one locale is, to me, a potentially good find for many people. Unity IAP manual and automated QA has caught some locale issues, and our developer users have found more, most recently with French locale and its comma-separator.

      Please connect with the Unity IAP Support (Forum https://forum.unity3d.com/forums/unity-iap.112/, Ticket https://support.unity3d.com/hc/en-us/requests/new?ticket_form_id=65905) team to help diagnose and resolve this. Any details which lead to reproducing and fixing this “in-the-wild” case may also help improve analytic dashboard accuracy, for better reporting and planning.

      -Nick, Unity IAP

  10. This is really cool, but you should make a game with the intention of selling it, a multi-year project that has some serious personal investment in it, ala-Epic/Paragon.

    You’ll find many more bugs and engine shortcomings when you make a full investment into a premium product.

    1. Can’t agree enough. Epic was/is constantly fixing bugs and adding features based on real world needs that arose from actual development.

  11. While I feel this is still pretty useful and am glad it’s being done, it’s testing the areas of Unity which Unity is best at, not the areas where it really falls down. Most of Unity’s major failings happen at a scale that cannot be achieved with a small prototype games. Things like Asset Bundles, large team workflows, multiplatform scalability, etc, are not going to be tested at this scale.

    1. I was going to write pretty much the same, then saw this comment.

      I believe creating prototype like projects isn’t so much going to reveal issues that professional game studios run into when aiming for high quality game experiences.

      The same applies to content. If you don’t try to create, for example, top-notch particle effects, you are not going to figure out what problems there might be to achieve such task.

      We all know that Unity is good at putting together prototypes, you proofed that many times and I’m convinced. It’s about time to think bigger.

  12. I hope you also try to upgrade past scenarios to new engines to find out what breaks after updates.

  13. I hope one day you make RTS tutorial :)

  14. Thanks, @CLAUS PETERSEN

    It’s nice to see you guys use this testing method. I hope you will do more of the multi-week scenario testing. Not only will this allow you to increase the depth in which you test targeted features, but the scope as well. As you indicated, you’re not going to be able to approach the scope of a real-world project, but it will allow you to cover very real-world scenarios that appear to be surprisingly absent. For example…
    Multiplayer networking is crucial for the majority of FPS games – and many racing games. It would be nice to see networking added to the scope of those projects.
    It appears that scenarios with networking is absent from all platforms except Desktop.
    WebGL didn’t appear as a target in any of the displayed scenarios.
    Having one scenario that crosses common platform targets like iOS, Android and WebGL would yield lots of exposed bugs.