Search Unity

A/B testing takes away the guesswork from the process of improving games.

In games, at any one time, countless variables have an impact on each player’s experience. That variability has a knock-on effect and makes optimization a much more complex undertaking than it might seem.

If you want to improve any aspect of your game, you need technology that isolates the aspect in question and tests your “improvements” effectively. If rigorous A/B testing bears out your hypothesis in specific conditions, on a subset of your total audience, you can roll it out on a larger scale.

What is an A/B Test?

An A/B test essentially compares different versions of the same variable to determine the impact of changes to that variable. The idea is to run concurrent tests in which conditions are as close to identical as possible, save for the variable being tested.

The control group continues to experience the “default” behavior and the treatment group receives the new behavior, so that you can compare the results and determine the impact of changing your chosen variable.

Why A/B test?

Testing changes in this way is important for several reasons:

  • By testing changes on a small portion of your population, you minimize any negative impact of rolling out a change.
  • You can statistically prove the effects of your proposed change, instead of relying on gut feeling.
  • By deciding on what constitutes “significant” change before running the test, you can be objective in your assessment.

How to perform a successful A/B test

Testing is an iterative process. The diagram below breaks down A/B testing step-by-step.

A few key factors are crucial to running a successful A/B test. Stick to the following rules, and you can’t go wrong:

Be patient

Patience and discipline are key to running a successful A/B test. Ending a test early, because you perceive a significant outcome appearing, will not give the best results. Wait for the test to run its full course.

Maintain consistency

You can only draw reliable conclusions from tests that action one change at a time. As soon as multiple variables are at work, you can’t accurately evaluate the precise impact of any of them. It’s one at a time, or not at all.

Always use the same audience

Your control and treatment groups must be made up of identical audiences, or the whole exercise is pointless. Geolocation, demographic, and play style are useful markers for segmentation because such things differentiate player profiles. If you want to know the true impact of a change, it must be tested on identical subjects.

A/B tests with Unity

We already have A/B testing functionality within the deltaDNA platform, and it is an integral part of optimizing everything – from game difficulty to in-app purchase (IAP) offer value. With cross-platform and rich data capability, this end-to-end solution enables publishers and developers to better understand different player behaviors and create personalized experiences, targeting individual players in real-time. Get a full-feature 30-day free trial of deltaDNA to discover what automated A/B testing can do for your game!

6 replies on “How A/B testing can help you optimize your mobile game”

With such a vast number of apps present in the app market, it is essential to optimize the app to stand out from the crowd.

I’d rather that Unity supported A/B testing with Remote Config and Unity Analytics. It’s highly disappointing that what was so easy with Remote Settings – now deprecated – now involves transitioning to a new analytics platform. Lack of coherent strategy, unfortunately such a familiar story.

these are not games, don’t call it games
these are purposefully built machines aimed at extracting value out of humans (or what they really are in freemium – products)
i hope civilization will come up with better term for these pieces of interactive software machinery – such as ‘unites’ – , but games ain’t it

They are called “Monetisation engines”. They come in many forms, also in the forms of games.

Comments are closed.