Audio is a difficult domain and area to test, both manually and in automation, for two main reasons:
People with a wide variety of backgrounds and degrees of knowledge of the audio domain will be using Unity’s new Audio Mixer, for example, audio programmers building their own customized systems and sound artists or composers who aren’t necessarily game audio integration or game production workflow experts.
Having an academic background in Musicology and a professional background in the computer games industry as an audio designer, I am familiar with some of our users’ backgrounds. So, as a Tester, I have a fair idea of audio designers/composers references to other audio tools and workflows when working in Unity.
One of the challenges we face in Unity 4 is that audio designers/composers who do not have a programming background are dependent on an audio programmer to create customized components and tools so they can tweak sounds in the scene at runtime. Or, they have to invest in a third party audio middleware solution that works with Unity.
With the Audio Mixer feature and its sub-features such as Snapshots/DSP Plugins/Duck Volume (just to mention a few) the Unity 5 editor introduces new concepts for audio designers/composers. Audio designers can now work productively directly in the editor instead of being forced to use additional software.
During early development and feature testing, I had discussions with developers about the role of the audio designer. What are the biggest challenges and what kind of DAW’s (Digital Audio Workstations such as Logic, Nuendo, Ableton Live and etc.) do they work with? Being an audio designer myself, what are my favourite tools and why?
We agreed that at one end of the scale we have the persona of the audio designer who is used to doing field/studio recordings, and at the other end we have the audio designer who’s familiar with programming and/or builds his or her own synths and patches in a visual programming language such as Pure Data/Max. To accommodate this, the Audio Mixer feature had to:
As a tester, I pitched into the discussions with my domain knowledge – also providing examples of other types of non-commercial audio middleware plugins developed for Unity. We had discussions about the biggest barriers that I had experienced as an audio designer, what kind of DAWs I had worked with and what features or workflows of those DAWs I liked and why. The questions that arose when I started testing the Audio Mixer feature in the pre-alpha state include:
Mind mapping helps me get an overview of the way a user can interact with a system’s UI in order to achieve a goal. Below is a mind map of the Audio area in Unity, which the Audio Mixer is part of. The mind map is based on James Bach’s heuristic testing strategy model.
One of the testing methods that I learnt more about during a Black Box Testing course was the use of “oracle heuristics”. A brief definition:
“The point of an oracle is to help you decide whether a product's behavior is inappropriate and, if so, to help you explain persuasively to someone else why they should consider it inappropriate.”
When performing manual testing on the Audio Mixer, I was very much aware that my primary oracle heuristic was "Comparable Products". As a user of Ableton Live 9, I was using the user interface and workflows from this DAW as a reference when testing. But I was also aware that there are many other commercial DAWs on the market and that comparing to only one of them was somewhat limiting or biased.
Further on in the development phase where more features were added and the user interface improved, another oracle heuristic started to sneak into my mind: simplicity.
Some years ago, during a Hack Week for Unity developers, the overall theme for the Hack Week was simplicity. One way of interpreting this theme, was to think of ways in which we could improve the workflows in the editor. It could be simple additions to the editor – for instance, one of the features that came out of that Hack Week was the Add Component button in the Game Object Inspector.
With this in mind, I started to think about ways the Audio Mixer and its subfeatures could make life easier for audio designers. I also asked myself how we should define what simplicity is in the context of the Audio Mixer?
Do testers and developers think of simplicity in the same way?
As a result one parameter I decided to measure was how many steps a user had to go through to be able to hear a sound in a scene. I created the flowchart below, which is based on a user scenario where the user creates an Audio Mixer, routes the Audio Source through the Mixer and hears a sound when playing the scene. With my "Comparable Product" Ableton Live, I go through 2 steps:
1. Drag a sample to a mixer channel
2. Hit Play - and I get an instant feedback when hearing the sound.
As I started to draw the diagram, I realized that with the pre-alpha version of the Audio Mixer I had to go through 6 steps before I got an audio feedback on my interaction in the editor. Is this desirable for an audio designer? With the diagram in hand, I could ask the developers for more information.
Working with oracle heuristics and visual tools, revealed further questions that helped me investigate the feature in depth. Some questions could not be answered though. This was partly because after a long period of testing a feature, you:
To avoid my own biases and to investigate usability issues further, I teamed up with User Experience Designer Stine Munkesø Kjærbøll and ran three rounds of usability testing sessions on audio designers/composers from the local game industry in Copenhagen. Some of the exciting things that we found that made a difference for our users were:
In an upcoming blog post the User Experience Team will elaborate further and take you through Usability Testing on the Audio Mixer feature. Stay tuned!
Is this article helpful for you?
Thank you for your feedback!