Search Unity

(or what I did at my first Hack Week with Unity)

This post is about a little trick for breaking infinite loops in scripts in Unity. It works with Windows / 64 bit in the Editor and in 64 bit builds with script debugging enabled. With a little more effort it can be made to work on 32 bit and even builds with script debugging disabled.

Infinite loops seems to be something that should be easily avoidable. But from time to time, I’ve encountered them in sneaky variants. Once it was the broken random function returning 1.000001 on impossibly rare cases. Another time a degenerate mesh sent a NaN right into an unsuspecting while(1) { d += 1.0; if(d>10.0) break; /* .. */ } loop. And then there was the broken data structure traversed by an algorithm that assumed current = current.next; would surely lead to an end eventually.

If you have experienced an infinite loop in your script code in Unity, you know it is quite unpleasant. Unity becomes unresponsive and you may have to kill the Editor entirely to get out of the mess. If you were lucky enough to have the debugger attached before running your game, you may be able to break it. But usually only if you can guess the right place in the code to set a breakpoint.

Some years back, before joining Unity, I found a way to break a script that is stuck like this. But it was not until my first Hack Week here, that I got to talk to the right people and realized why the trick works and how it may be used to create a proper way to break scripts in Unity (sneak peek of my Hack Week project). Until we get that feature properly done and release it, you can use the trick below. Or just hang on for the fun of a little tour into disassembled, jit’ed code. What could possibly go wrong?

Do not try this at work!

As a trained professional you know the value of practice, so try this out on a toy project before you attempt a rescue operation at work. Fire up Unity and create an empty project, add a box to an empty scene and create a new C# script “Quicksand” attached to the box. The script should contain this code:

Now hit play and click the box. Observe Unity freeze up and experience the onsetting rush of panic until you remember that this is just a test. No actual work is going to be harmed!

So now your script is stuck and Unity seems to be hung. Let us start up a new instance of Visual Studio.

For this to work you probably (I didn’t check to be honest) need to have selected C++ as one of the languages when you installed VS. Go to Debug menu and select Attach to Process (NOTE: this is not the same option you usually choose for attaching to Unity). Locate the Unity process and attach to it.

img_02

Having attached the debugger to the stuck Unity, select “Debug | Break all” and find the disassembly view showing the code currently executing on the main thread. The following gif shows the little dance you have to perform. Perhaps you even need to click on “show disassembly” or something like that depending on your configuration of Visual Studio. (On one machine where I tried it, I had to hit F10, which does a single-step, for the disassembly to show).

img_03

As you maybe know, the scripts are — for performance reasons — translated into machine code on the fly before being executed. This is known as jit-compiling (just-in-time compiling). The result is what you see in the disassembly window. Hopefully it looks something like this:

img_04

In this case you can almost see the infinite loop (indicated by the red arrow I have artistically rendered on top of the screen snip). There is one mov, one cmp and a lot of nop’s and then the jmp loops right back to where we started. No escape!

In a more realistic case, your C# code will be more complicated, and it will be harder to tell what is going on, but you don’t really have to understand it, because here comes the trick: hit F10 (single step) a number of times until you get to one of the “cmp dword ptr [r11], 0″ instructions. They should be sprinkled liberally all over the code because they are part of the debugging infrastructure. After a few steps, you may end up with something like this:

img_05

With a bit of luck you already have a window named “Autos” (if not, use Debug | Windows | Auto to find it). It should show you the value of the registers that are in play at this point:

img_06

Now simply change the value of R11 to 0. Like this:

img_07

If you were to execute the cmp instruction now, it would try to read from address 0 which is going to generate an exception. And that is actually exactly what we want: So hit F5 (continue program execution) and answer “Continue” to the dialog box that pops up:

img_08

If all went well, you now get a proper (Mono-)exception in Unity, the loop is broken and Unity is back to normal. You can save your work and look at the call stack in the Console to see which part of your script code caused the hang.

img_09

That’s it. Go forth and loop! A little warning is in place: you have now messed around pretty deep inside Unity and it is prudent to save your work (if needed) and restart the Editor. My experience is that everything seem quite healthy, but just to be on the safe side.

Why does this hack work?

The reason this works at all is that Mono has a built in system for debugging scripts. It works by sprinkling (actually, once for each C# line) the jit-code with reads from a specific memory address. That is the “cmp dword ptr [r11], 0” instructions we saw above. When you are in debugging mode and are single stepping through your code, the page that holds this memory address is made read-only and we will hit an exception once for every C# line of code. The Mono framework can catch this outside of the jit-ed code and basically pause the execution after every line.

The trick we did above, where we set the register r11 to be 0, will end up generating the same type of exception, because the address 0 is never readable. So the debugging framework sees something that looks like single-stepping, but because we are not really debugging, a NullReferenceException is generated and we get a nice call stack. Very convenient!

This technique also works for standalone games. You attach to yourgame.exe, break all, find the jit code, force a memory fault and you should be good. You will have to look up the call stack in the log file, though.

Corner cases

The example we just looked at was, to put it mildly, conveniently simplified for tutorial purposes! In reality, there are a number of catches you may run into. When you hit “break all” you may not actually break into ‘clean’ jit’ed code. If your C# code makes use of any API calls, the program might be inside some of the core Unity code. It will look like this:

img_10

Here we see the program having called into GetPosition. When the top of the Call stack contains real function names and not just hex addresses, it is usually a sign we have left mono / jit’ed code. But just hit Shift-F11 (Step Out) a few times until you are back in jit-land (nop’s galore is also a good indicator of jit-code).

Sometimes you can manage to break Unity at a point where the main thread is not active. It is probably easiest just to continue (F5) and then break all again until the main thread is active.

There are probably more weird cases, but hey, this is debugging, so improvisation is key!

What about 32bit

You can do something similar in 32bit mode. The jit-code looks a little different. Perhaps something like this:

img_11

This means we read from 0xB10000 in this case. To provoke a page fault, you need to actually change the code because the address is hardcoded directly in the instruction and not in a register as with 64 bit. So you open a memory view (Debug | Windows | Memory | Memory1) and navigate to the address of the instruction (the yellow arrow), that is, 0x65163DC in our case. Here we find:

img_12

You can recognize the address: it is the “b1” found 4 bytes in from the start. Change it to 00 and then continue (F5) as before. This will have the same effect, but with the difference from the 64 bit case, that you will break out every time you hit this location.

So what about non debug mode?

Ok if you are really unlucky, you may have a bug that is only reproducible when you compile your scripts with debugging disabled. In this case you have to improvise a bit. If you can look at the code and find a way to provoke a read fault you should be golden, but it may or may not be super easy. As a last resort you may need to inject, manually, something like cmp eax, dword ptr ds:[0x0] which we know from above happens to be the sequence 3b 05 00 00 00 00. But maybe we are a bit more lucky. Let us try our example script from above. Breaking it yields:

img_13

Oh no! The worst. The compiler optimized it to just one jmp instruction looping on itself. There isn’t even room for adding in our cmp (the jmp is relative and only takes up 2 bytes). However, since we are assuming everyone is desperate (a release build hung, after all) we don’t have to be too careful and can just trash the code with our read. Navigate to 4D34446 in the memory window and fill in 3b 05 00 00 00 00 on top on whatever is there. Hit continue (F5) and hope. In my case the game (I was doing this part of the test with a standalone game)  came back to life and I could inspect the output log to find:

img_14

At this point you really should shut down the game as you have effectively ruined a part of the jit generated code and your scripts will likely not work anymore. But at least you know where you were stuck.

Sometimes you can find a read instruction in the vicinity of where you stopped. Then you can right click on it, select “Set next statement” and by setting a register to 0 you may be able to create the right exception that way.

Conclusion

So with a bit of trickery it seems unbreakable loops are in fact breakable. Hurry up and try it out so you can join the ranks of grizzled veterans growling “Ha! In my days we did it in disassembly!”. Soon we’ll ship a better solution and it will be forever too late!

21 Comments

Subscribe to comments

Comments are closed.

  1. Anselmo Fresquez

    May 29, 2016 at 7:42 am

    I like infinite loops. They remind me that I am human.

    1. Lmao Such a true answer!

  2. Hmmm…there is a SimpleWebServer code that we had a bug in and I couldn’t attach to it a normal way via a breakpoint in the SimpleWebServer code because the client would make a request to the Unity httpdlistener but then that listener would never return to the SimpleWebServer code.

    How to attach to that listen in Windows 10? Identify the ports connected with netstat and then how to find the Windows process using that port? And then attach VS debugger to the process?

  3. What a waste of time to research such a thing *facepalm*.
    There are a lot of issues in the Project and than a blog post on this useless stuff, well i don’t get it.

    Thanks for nbothing.

    1. stop b*tchng

  4. Very helpful post, thank you. That will surely come in handy when debugging and is a lot better than littering the code with “I was here” log statements to narrow down freezes.

    And yes.. can’t wait for the panic button built in into the editor for the “common case” of a simple infinite loop fubar.

    Now what about StackOverflows? In Editor-code, they tend to immediately quit the process without any stack dump or anything in the log. In play-mode within the editor, they basically freeze the editor while trying to pump out a million log entries per frame or something like that. Would be nice to have some tooling / tricks in the belt for these kind of errors too.. ;)

    (Or in general: Is there any (assembly-) trick to deactivate the log, so if an editor runs havoc and want to log millions of entries per update, to calm him down for some seconds to be able to save the scene/assets?)

    1. Peter Andreasen

      June 2, 2016 at 9:00 am

      Hmm. Can you show a small example of what you mean? My trivial stackoverflow code looks like this:

      ..
      void Update() { Update(); }

      and afaict, while it spews out StackOverflowException in play mode, I am able to simply stop it? You must have another case in mind?

  5. Robin Fischer

    May 25, 2016 at 3:30 pm

    Nice post! This can come handy in sticky situations where I want to keep my EditorWindows states, etc.

    But I think if you are just after the scene data there is a much simpler way of recovering that. Every time you press on the Unity play button. Unity copies the active scene(s) to the “Temp\__Backupscenes” directory. If Unity leaves the play mode it loads the backup in that directory. This is the way Unity can “reset” your changes done while playing. If you terminate Unity in playmode the “Temp” directory is still there and you can just copy the scenes you worked on into the Assets directory.
    When you open Unity it will clear the backup directory. So copy the backups before restarting Unity.

    Hope this helps :)

  6. Why not just make unity save the current scene and project before running anything (and have an option to disable this behavior for people who don’t want it).

    That way even if the editor does crash you won’t lose your work. It is also more consistent with other developer tools.

    1. We did this in our project already, using EditorApplication.playmodeStateChanged:


      [InitializeOnLoad]
      public static class SaveBeforePlay
      {
      static SaveBeforePlay()
      {
      EditorApplication.playmodeStateChanged += OnPlayModeChanged;
      }

      static void OnPlayModeChanged()
      {
      if (!EditorApplication.isPlaying && EditorApplication.isPlayingOrWillChangePlaymode)
      {
      EditorSceneManager.SaveOpenScenes();
      AssetDatabase.SaveAssets();
      }
      }
      }

  7. Michael Chugg

    May 24, 2016 at 7:17 pm

    Very handy in those cases where you forgot to save your scene and are working with a while loop that accidentally gets out of hand. I will be referring to this page again once that happends to me again. (Hopefully never but you know…)

  8. Or just kill unity and restart without all this pain.

    1. Well the pain can be greater if you have a lot of unsaved work!

    2. This is way too much trouble. I’m not worried about losing work because I save my scene so often. And I don’t use any SCCS because I think they are all unstable and prone to error. I just save my complete project under a date-stamped name before I make any change. After a day of coding I’ll have 20 copies of my project. And I use a diff app if I need to find changes.

    3. Peter Andreasen

      May 25, 2016 at 1:15 am

      Others have already mentioned the scenario where you have unsaved work. But equally important, I think, is that you have to find and fix the code that is causing the loop. That can be pretty tricky if you have a lot of code and the loop is rare/random. Using this method at least you get a call stack straight to where the problem is.

  9. Bryan Livingston

    May 24, 2016 at 6:33 pm

    Shouldn’t this be something the Unity editor should handle?

    It’s pretty ridiculous that the editor doesn’t have some kind of break button and won’t even automatically save your scene changes before running. I’ve lost changes in the editor before by accidentally hitting an infinite loop, which is really unacceptable for a $1500 product.

    1. It literally says in the post that they’re about to ship a solution built into Unity.

  10. Ugly, you wanna do simple stuff?, well you can, but with the ugliest process.

  11. Raziel Anarki

    May 24, 2016 at 4:14 pm

    can’t wait for the better solution! will it be an instruction counter? :)