Search Unity

Breakout: How to stop an infinite loop in a Unity C# script

May 24, 2016 in Engine & platform | 10 min. read
img_02
img_02
Topics covered
Share

Is this article helpful for you?

Thank you for your feedback!

(or what I did at my first Hack Week with Unity)

This post is about a little trick for breaking infinite loops in scripts in Unity. It works with Windows / 64 bit in the Editor and in 64 bit builds with script debugging enabled. With a little more effort it can be made to work on 32 bit and even builds with script debugging disabled.

Infinite loops seems to be something that should be easily avoidable. But from time to time, I’ve encountered them in sneaky variants. Once it was the broken random function returning 1.000001 on impossibly rare cases. Another time a degenerate mesh sent a NaN right into an unsuspecting while(1) { d += 1.0; if(d>10.0) break; /* .. */ } loop. And then there was the broken data structure traversed by an algorithm that assumed current = current.next; would surely lead to an end eventually.

If you have experienced an infinite loop in your script code in Unity, you know it is quite unpleasant. Unity becomes unresponsive and you may have to kill the Editor entirely to get out of the mess. If you were lucky enough to have the debugger attached before running your game, you may be able to break it. But usually only if you can guess the right place in the code to set a breakpoint.

Some years back, before joining Unity, I found a way to break a script that is stuck like this. But it was not until my first Hack Week here, that I got to talk to the right people and realized why the trick works and how it may be used to create a proper way to break scripts in Unity (sneak peek of my Hack Week project). Until we get that feature properly done and release it, you can use the trick below. Or just hang on for the fun of a little tour into disassembled, jit'ed code. What could possibly go wrong?

Do not try this at work!

As a trained professional you know the value of practice, so try this out on a toy project before you attempt a rescue operation at work. Fire up Unity and create an empty project, add a box to an empty scene and create a new C# script “Quicksand” attached to the box. The script should contain this code:

using UnityEngine;
public class Quicksand : MonoBehaviour {
   void OnMouseDown()

   {
       while(true)

       {

           // "Mind you, you'll keep sinking forever!!", -- My mom

       }

   }

}

Now hit play and click the box. Observe Unity freeze up and experience the onsetting rush of panic until you remember that this is just a test. No actual work is going to be harmed!

So now your script is stuck and Unity seems to be hung. Let us start up a new instance of Visual Studio.

For this to work you probably (I didn't check to be honest) need to have selected C++ as one of the languages when you installed VS. Go to Debug menu and select Attach to Process (NOTE: this is not the same option you usually choose for attaching to Unity). Locate the Unity process and attach to it.

img_02

Having attached the debugger to the stuck Unity, select "Debug | Break all" and find the disassembly view showing the code currently executing on the main thread. The following gif shows the little dance you have to perform. Perhaps you even need to click on “show disassembly” or something like that depending on your configuration of Visual Studio. (On one machine where I tried it, I had to hit F10, which does a single-step, for the disassembly to show).

img_03

As you maybe know, the scripts are -- for performance reasons -- translated into machine code on the fly before being executed. This is known as jit-compiling (just-in-time compiling). The result is what you see in the disassembly window. Hopefully it looks something like this:

img_04

In this case you can almost see the infinite loop (indicated by the red arrow I have artistically rendered on top of the screen snip). There is one mov, one cmp and a lot of nop’s and then the jmp loops right back to where we started. No escape!

In a more realistic case, your C# code will be more complicated, and it will be harder to tell what is going on, but you don’t really have to understand it, because here comes the trick: hit F10 (single step) a number of times until you get to one of the "cmp dword ptr [r11], 0" instructions. They should be sprinkled liberally all over the code because they are part of the debugging infrastructure. After a few steps, you may end up with something like this:

img_05

With a bit of luck you already have a window named “Autos” (if not, use Debug | Windows | Auto to find it). It should show you the value of the registers that are in play at this point:

img_06

Now simply change the value of R11 to 0. Like this:

img_07

If you were to execute the cmp instruction now, it would try to read from address 0 which is going to generate an exception. And that is actually exactly what we want: So hit F5 (continue program execution) and answer “Continue” to the dialog box that pops up:

img_08

If all went well, you now get a proper (Mono-)exception in Unity, the loop is broken and Unity is back to normal. You can save your work and look at the call stack in the Console to see which part of your script code caused the hang.

img_09

That’s it. Go forth and loop! A little warning is in place: you have now messed around pretty deep inside Unity and it is prudent to save your work (if needed) and restart the Editor. My experience is that everything seem quite healthy, but just to be on the safe side.

Why does this hack work?

The reason this works at all is that Mono has a built in system for debugging scripts. It works by sprinkling (actually, once for each C# line) the jit-code with reads from a specific memory address. That is the "cmp dword ptr [r11], 0" instructions we saw above. When you are in debugging mode and are single stepping through your code, the page that holds this memory address is made read-only and we will hit an exception once for every C# line of code. The Mono framework can catch this outside of the jit-ed code and basically pause the execution after every line.

The trick we did above, where we set the register r11 to be 0, will end up generating the same type of exception, because the address 0 is never readable. So the debugging framework sees something that looks like single-stepping, but because we are not really debugging, a NullReferenceException is generated and we get a nice call stack. Very convenient!

This technique also works for standalone games. You attach to yourgame.exe, break all, find the jit code, force a memory fault and you should be good. You will have to look up the call stack in the log file, though.

Corner cases

The example we just looked at was, to put it mildly, conveniently simplified for tutorial purposes! In reality, there are a number of catches you may run into. When you hit “break all” you may not actually break into ‘clean’ jit’ed code. If your C# code makes use of any API calls, the program might be inside some of the core Unity code. It will look like this:

img_10

Here we see the program having called into GetPosition. When the top of the Call stack contains real function names and not just hex addresses, it is usually a sign we have left mono / jit’ed code. But just hit Shift-F11 (Step Out) a few times until you are back in jit-land (nop’s galore is also a good indicator of jit-code).

Sometimes you can manage to break Unity at a point where the main thread is not active. It is probably easiest just to continue (F5) and then break all again until the main thread is active.

There are probably more weird cases, but hey, this is debugging, so improvisation is key!

What about 32bit

You can do something similar in 32bit mode. The jit-code looks a little different. Perhaps something like this:

img_11

This means we read from 0xB10000 in this case. To provoke a page fault, you need to actually change the code because the address is hardcoded directly in the instruction and not in a register as with 64 bit. So you open a memory view (Debug | Windows | Memory | Memory1) and navigate to the address of the instruction (the yellow arrow), that is, 0x65163DC in our case. Here we find:

img_12

You can recognize the address: it is the “b1” found 4 bytes in from the start. Change it to 00 and then continue (F5) as before. This will have the same effect, but with the difference from the 64 bit case, that you will break out every time you hit this location.

So what about non debug mode?

Ok if you are really unlucky, you may have a bug that is only reproducible when you compile your scripts with debugging disabled. In this case you have to improvise a bit. If you can look at the code and find a way to provoke a read fault you should be golden, but it may or may not be super easy. As a last resort you may need to inject, manually, something like cmp eax, dword ptr ds:[0x0] which we know from above happens to be the sequence 3b 05 00 00 00 00. But maybe we are a bit more lucky. Let us try our example script from above. Breaking it yields:

img_13

Oh no! The worst. The compiler optimized it to just one jmp instruction looping on itself. There isn’t even room for adding in our cmp (the jmp is relative and only takes up 2 bytes). However, since we are assuming everyone is desperate (a release build hung, after all) we don’t have to be too careful and can just trash the code with our read. Navigate to 4D34446 in the memory window and fill in 3b 05 00 00 00 00 on top on whatever is there. Hit continue (F5) and hope. In my case the game (I was doing this part of the test with a standalone game)  came back to life and I could inspect the output log to find:

img_14

At this point you really should shut down the game as you have effectively ruined a part of the jit generated code and your scripts will likely not work anymore. But at least you know where you were stuck.

Sometimes you can find a read instruction in the vicinity of where you stopped. Then you can right click on it, select “Set next statement” and by setting a register to 0 you may be able to create the right exception that way.

Conclusion

So with a bit of trickery it seems unbreakable loops are in fact breakable. Hurry up and try it out so you can join the ranks of grizzled veterans growling “Ha! In my days we did it in disassembly!”. Soon we’ll ship a better solution and it will be forever too late!

May 24, 2016 in Engine & platform | 10 min. read

Is this article helpful for you?

Thank you for your feedback!

Topics covered
Related Posts