The platoon of manual testers at Unity – Software Test Engineers team
Hello everyone, my name is Agnieszka Loza, I’m one of the QA Leads at Unity. I’d like to present to you the Software Test Engineers team (STE). If you are interested in how our army of testers works, you may have already seen the articles about the SDET and Toolsmith teams from Elvis and Kasper. Or maybe you have read my previous post about the STE team offsite.
The STE team is a sub team of our QA department. There are 20 testers in our team, working from Unity offices in Copenhagen, Odessa, Vilnius, Montreal, Boston, Berlin, and a few more locations. We are, what you may call, manual testers but there is much more to it than that. In this post, I’d like to tell you about the responsibilities we have and the activities we carry out. I will tell you a story about how STE team contributes to the never ending battle for better quality!
Responsibilities and challenges
Unity is a big and complex software. In order to make it more testable, we divided it into areas – every area is a distinct chunk of functionality. To give a few examples, 2D is an area in Unity, so are: Animation, Terrain, Asset Import, Particle System, PS4, Xbox, UI, Shaders and so on. All in all, there are around 70 areas. Every area has a designated STE and at least one backup STE who are responsible for planning and carrying out all the necessary test activities related to it.
To be able to create and execute a test strategy for an area, STEs have to be knowledgeable on many fronts. They must know about Unity – especially the area they work on, but also about the ways the area’s functionality interacts with the rest of the engine. They need to know test techniques and how to properly apply them to retrieve the information they’re looking for. Technical knowledge is also required of an STE – they don’t always code on their own, but they need to understand basic programming concepts and be able to do some coding, when that’s required. Finally, knowing about the game industry is also a big advantage. Knowing how game developers use Unity and the expectations they have of the engine is a great source of testing inspiration.
Testers take all these considerations into account and create a test strategy defining what needs to be tested and how to test it. They use what they know and draw a map – a plan of how they will look for landmines (read: bugs). They try to figure out what they don’t know and scout around to retrieve missing information.
STEs are active from the early stages of developing a new feature. They work closely with developers of their areas and are the first ones to look at the feature in progress – long before it goes into a release or reaches the alpha group. The main task of a tester is to become familiar with the new features, consider related risks and provide comprehensive feedback such as suggestions for improved workflows, discovered bugs and comments on learnability etc. The Test Engineer’s job is to provide an initial evaluation of the quality that will help to make the call whether the feature is ready to be included into a release or not.
Once the feature is in a release, more elaborate testing continues. STEs research all the possible risks related to new features and based on this analysis they test, focusing on the functionality that may be shaky or yet unexplored. Testers work in pairs. They use more challenging test data and look at complex user scenarios and integration with other features. It is also a good way to ensure we have a fresh pair of eyes on most new features. During this phase, testers strive to uncover all the critical problems.
Based on the testing done in the alpha phase, STEs create manual test cases that will be used later on, in beta and release candidate phases, to catch possible regressions. Even though the main focus of these phases is on polishing the features and on bug fixes, we still run the risk that the code changes will cause regressions. Every STE is designing and creating the regression test suites for their areas, making sure that the coverage is sufficient, and that the manual test cases do not overlap with automation testing – which would be redundant.
Other testing activities include Release Acceptance Testing (RAT) and bug verification. RAT is a very narrow set of manual regression tests focused on detecting critical bugs in the builds that we are about to release – either to the public or to the alpha and beta users. Bug verification is a frequent area-grooming activity – testers verify fixed bugs, they investigate those that were deferred and close the ones that are no longer reproducible. Knowing the bug situation for the area is one of the metrics we use to evaluate the quality of the area.
During all three stages of a release – alpha, beta and release candidate – testers report back from the testing front. There is a lot of interesting stories a tester can tell. What is the status of a given feature? Is it ready to be used? How does the feature perform under various scenarios? Is the feature doing what is supposed to do? Is it compatible with all related functionality? What do the users think of it? What is the number of known bugs? What’s the trend in regressions? How severe are the problems the feature still has? All these different questions – and more – are answered by the STEs in the area reports they send out thrice per release.
Who reads these reports? Everyone! Developers, release managers, testers, test managers, release coordinators, executive team – everyone who has the interest in the quality of the software or has to make decisions regarding development and shipping of Unity.
STE Team provides a unique service. We collect and present the feedback about the quality of the Unity engine so that everyone can be better informed in making decisions. Our goal is to support Unity Technologies and we do so to the best of our abilities and available resources.
The team continually seeks to learn new skills and improve the testing toolbox to support the goal stated above. We go to various courses, training sessions and conferences. We share knowledge within the group on a weekly basis and we strive to improve our processes.
This was a summary of how the STE team stays busy – by testing, reporting and learning – all oriented around supporting Unity Technologies in making strategic business decisions. If you find this tester profile interesting feel free to contact us! We are hiring!