In a previous blog post, “Testing the Audio Mixer”, the Audio Mixer was discussed from a Software Test Engineering point of view. In this blog post we will pick up where we left off: involving real users in evaluating a system in development.
In the Summer and Fall of 2014, we, Stine Kjærbøll (User Experience Team Lead) and Nevin Eronde (User Experience Researcher), ran three rounds of usability testing sessions with audio designers/composers from the local games industry in Copenhagen.
The users that we recruited for usability testing had backgrounds in games and art installations. They were:
- Audio programmers
- The “One-Man Army» (programmer/artist/audio designer)
- Sound designers
- Audio-visual artists
Think Aloud Protocol – Involving real users
The Think Aloud Protocol is a user research method that involves test subjects thinking aloud as they are performing a set of specified tasks. The test subjects are encouraged to say whatever they are thinking, doing and feeling as they are performing the tasks.
We wanted to perform the Think Aloud test was for several reasons:
- The Audio Mixer is designed for a target audience that ranges from people who are programming experts and people who have never even opened a script. Would those audio designers, who are not comfortable with scripting, be able to use the Audio Mixer without the the help of a programmer? To what extent?
- Having an Audio Mixer inside of a game development tool is a new concept in itself. Would users understand the concepts of Snapshots, Ducking Volume, Audio Groups and Audio Sources?
- Unity editor workflow with references from 3rd party DAW (Digital Audio Workstation) tools. Would the Audio Mixer and its concepts be easy to use for an audio designer that is working with other types of DAWs?
- Would they understand the concept of tweaking audio effects at runtime?
All of our user research is confidential, but this Kids React video is pretty close to how a Think Aloud session can look like:
Biases during development
Another reason why we conducted Think Aloud Protocols was to avoid our own biases. Whenever you are testing, researching and designing a feature, you can not avoid biases, but you can be aware of them.
Some of the biases we were aware of with regards to the Audio Mixer as well as the users that we recruited from the local games industry were:
- Test Subjects were working on smaller computer game productions
- We could not invite audio designers working on large AAA-scale games,since such studios are scarce to come by in the Copenhagen area
- The QA testing that had been done on the Audio Mixer was with specific DAWs in mind
- Lastly, our own familiarity with the Audio Mixer meant that we were biased with regards to naming conventions and how the system functioned.
This is also where usability testing gets interesting! With the above research questions and biases in mind, we started designing the test.
Designing the Test
When designing usability tests, you always have to make a decision whether to go for realism, abstraction or something in between. If we had designed a test that used a large Unity project, such as Bootcamp or Angry Bots, that could count as a realistic test scenario, but the complexity of the project might be overwhelming for a test subject unfamiliar with the project. S/he would end up spending a lot of time familiarizing themselves with the project instead of focusing on the Audio Mixer. The Audio Mixer itself is an extensive system, so we decided to go for a minimalist test setup, to avoid any added complexity.
We used a scene with a Red Cube and Blue Cube. All the sound files that were needed were provided in a folder. Not a real life scenario, but the complexity is removed, so the test subject could focus on the tasks.
The tasks of the test subjects were to:
- Build up an ambience in the scene
- Trigger Snapshots
- Trigger Ducking Volume
- Explore the Audio Mixer effects and Send/Receive units
Lessons learned during Think Aloud Protocol
When conducting Think Aloud Protocols, it is always an interesting study in interaction design. We learned many lessons and below are a few examples:
What you think is “obvious” is not obvious to the user
The “Edit in Play Mode” button is a very useful feature for audio designers, as it enables them to tweak and adjust sounds at runtime. In a very eary prototype, the button was labelled as “Edit Snapshots”. Technically, this is what you are doing when enabling the button and adjusting the Audio Mixer settings at runtime. The problem was that the test subjects did not interpret the “Edit Snapshots” button as a natural part of their workflow. They ignored the button, simply because they did not intuitively associate it with the ability to edit the Audio Mixer settings at runtime. The button was then renamed to “Edit in Play mode”. We ran a new round of Think Aloud tests and from those tests we could verify that it was more intuitive for our test subjects to enable the “Edit in Play mode” button, as they now interpreted the button label as a part of their workflow.
Simplicity versus complexity
When developing features we often think of how to make the feature simple to use for our users, whether the user is advanced or a new user. One thing we observed during usability testing of the Audio Mixer was that in some cases users wanted the complexity. In the first version of the Ducking Volume, the feature had very few parameters for the user to tweak. The test subjects expressed that “something was missing”. We learned that audio designers are actually used to the complexity of multiple parameters that they can adjust and tweak from the DAWs. A feature with very few parameters was suddenly a complex thing to understand.
Below is an early design example of the Ducking Volume.
After the feedback from our test subjects, the Ducking Volume was redesigned to a sidechained compressor with parameters and controls that the test subjects are used to from DAWs.
Getting to know you!
All the feedback that we get from our users is so valuable to us! We, the User Experience Team, want to get closer to you! Meet us at Unite Europe 2015 in Amsterdam and help us make Unity even better!