Serialization in Unity

June 24, 2014 in Technology

In the spirit of sharing more of the tech behind the scenes, and reasons why some things are the way they are, this post contains an overview of Unity’s serialization system. Understanding this system very well can have a big impact on the effectiveness of your development, and the performance of the things you make.  Here we go.

Serialization of “things” is at the very core of Unity. Many of our features build ontop of the serialization system:

  • Storing data stored in your scripts. This one most people are probably somewhat familiar with.
  • Inspector window. The inspector window doesn’t talk to the C# api to figure out what the values of the properties of whatever it is inspecting is. It asks the object to serialize itself, and then displays the serialized data.
  • Prefabs. Internally, a prefab is the serialized data stream of one (or more) game objects and components. A prefab instance is a list of modifications that should be made on the serialized data for this instance. The concept prefab actually only exists at editor time. The prefab modifications get baked into a normal serialization stream when Unity makes a build, and when that gets instantiated, the instantiated gameobjects have no idea they were a prefab when they lived in the editor.
  • Instantiation. When you call Instantiate() on either a prefab, or a gameobject that lives in the scene, or on anything else for that matter (everything that derives from UnityEngine.Object can be serialized), we serialize the object, then create a new object, and then we “deserialize” the data onto the new object. (We then run the same serialization code again in a different variant, where we use it to report which other UnityEngine.Object’s are being referenced. We then check for all referenced UnityEngine.Object’s if they are part of the data being Instantiated(). If the reference is pointing to something “external” (like a texture) we keep that reference as it is, if it is pointing to something “internal” (like a child gameobject), we patch the reference to the corresponding copy).
  • Saving. If you open a .unity scene file with a text editor, and have set unity to “force text serialization”, we run the serializer with a yaml backend.
  • Loading. Might not seem surprising, but backwards compatible loading is a system that is built on top of serialization as well. In-editor yaml loading uses the serialization system, as well as the runtime loading of scenes and assets. Assetbundles also make use of the serialization system.
  • Hot reloading of editor code. When you change an editor script, we serialize all editor windows (they derive from UnityEngine.Object!), we then destroy all the windows, unload the old c# code, load the new c# code, recreate the windows, and finally deserialize the datastreams of the windows back onto the new windows.
  • Resource.GarbageCollectSharedAssets(). This is our native garbage collector and is different to the C# garbage collector. It is the thing that we run after you load a scene to figure out which things from the previous scene are no longer referenced, so we can unload them. The native garbage collector runs the serializer in a mode where we use it to have objects report all references to external UnityEngine.Objects. This is what makes textures that were used by scene1, get unloaded when you load scene2.

The serialization system is written in C++, we use it for all our internal object types (Textures, AnimationClip, Camera, etc). Serialization happens at the UnityEngine.Object level, each UnityEngine.Object is always serialized as a whole. They can contain references to other UnityEngine.Objects and those references get serialized properly.

Now you may say that none of this concerns you very much, you’re just happy that it works and want to get on with actually creating some content. However, this will concern you, as we use this same serializer to serialize MonoBehaviour components, which are backed by your scripts. Because of the very high performance requirements that the serializer has, it does not in all cases behave exactly like what a C# developer would expect from a serializer. Here we’ll describe how the serializer works and some best practices on how to make the best use of it.

What does a field of my script need to be in order to be serialized?

  • Be public, or have [SerializeField] attribute
  • Not be static
  • Not be const
  • Not be readonly
  • The fieldtype needs to be of a type that we can serialize.

Which fieldtypes can we serialize?

  • Custom non abstract classes with [Serializable] attribute.
  • Custom structs with [Serializable] attribute. (new in Unity4.5)
  • References to objects that derive from UntiyEngine.Object
  • Primitive data types (int,float,double,bool,string,etc)
  • Array of a fieldtype we can serialize
  • List<T> of a fieldtype we can serialize

So far so good. So what are these situations where the serializer behaves differently from what I expect?

Custom classes behave like structs

[Serializable]
class Animal
{
    public string name;
}

class MyScript : MonoBehaviour
{
    public Animal[] animals;
}

If you populate the animals array with three references to a single Animal object, in the serializationstream you will find 3 objects. When it’s deserialized, there are now three different objects. If you need to serialize a complex object graph with references, you cannot rely on Unity’s serializer doing that all automagically for you, and have to do some work to get that object graph serialized yourself. See the example below on how to serialize things Unity doesn’t serialize by itself.

Note that this is only true for custom classes, as they are serialized “inline” because their data becomes part of the complete serializationdata for the MonoBehaviour they are used in. When you have fields that have a reference to something that is a UnityEngine.Object derived class, like a “public Camera myCamera”, the data from that camera are not serialized inline, and an actual reference to the camera UnityEngine.Object is serialized.

No support for null for custom classes

Pop quiz. How many allocations are made when deserializing a MonoBehaviour that uses this script:

class Test : MonoBehaviour
{
    public Trouble t;
}

[Serializable]
class Trouble
{
    public Trouble t1;
    public Trouble t2;
    public Trouble t3;
}

It wouldn’t be strange to expect 1 allocation, that of the Test object. It also wouldn’t be strange to expect 2 allocations, one for the Test object and one for a Trouble object. The correct answer is 729. The serializer does not support null. If it serializes an object and a field is null, we just instantiate a new object of that type and serialize that. Obviously this could lead to infinite cycles, so we have a relatively magical depth limit of 7 levels. At that point we just stop serializing fields that have types of custom classes/structs and lists and arrays. [1]

Since so many of our subsystems build on top of the serialization system, this unexpectedly big serializationstream for the Test monobehaviour will cause all these subsystems to perform more slowly than necessary. When we investigate performance problems in customer projects, we almost always find this problem and we added a warning for this situation in Unity 4.5.  We actually messed up the warning implementation in such a way that it gives you so many warnings, you have no other option but to fix them right away. We’ll soon ship a fix for this in a patch release, the warning is not gone, but you will only get one per “entering playmode”, so you don’t get spammed crazy. You’d still want to fix your code, but you should be able to do it at a time where it suits you.

No support for polymorphism

If you have a

public Animal[] animals

and you put in an instance of a dog, a cat and a giraffe, after serialization, you will have three instances of Animal.

One way to deal with this limitation is to realize that it only applies to “custom classes”, which get serialized inline. References to other UnityEngine.Object’s get serialized as actual references and for those, polymorphism does actually work. You’d make a ScriptableObject derived class or another MonoBehaviour derived class, and reference that. The downside of doing this, is that you need to store that monobehaviour or scriptable object somewhere and cannot serialize it inline nicely.

The reason for these limitations is that one of the core foundations of the serialization system is that the layout of the datastream for an object is known ahead of time, and depends on the types of the fields of the class, instead of what happens to be stored inside the fields.

I want to serialize something that Unity’s serializer doesn’t support. What do I do?

In many cases the best approach is to use serialization callbacks. They allow you to be notified before the serializer reads data from your fields and after it is done writing to them. You can use this to have a different representation of your hard-to-serialize data at runtime than when you actually serialize. You’d use these to transform your data into something Unity understands right before Unity wants to serialize it, you also use it to transform the serialized form back into the form you’d like to have your data in at runtime, right after Unity has written the data to your fields.

Let’s say you want to have a tree datastructure. If you let Unity directly serialize the data structure, the “no support for null” limitation would cause your datastream to become very big, leading to performance degradations in many systems:

using UnityEngine;
using System.Collections.Generic;
using System;

public class VerySlowBehaviourDoNotDoThis : MonoBehaviour
{
    [Serializable]
    public class Node
    {
        public string interestingValue = "value";
        
       //The field below is what makes the serialization data become huge because
       //it introduces a 'class cycle'.
       public List<Node> children = new List<Node>();
    }

    //this gets serialized
    public Node root = new Node();

    void OnGUI()
    {
        Display (root);
    }

    void Display(Node node)
    {
        GUILayout.Label ("Value: ");
        node.interestingValue = GUILayout.TextField(node.interestingValue, GUILayout.Width(200));

        GUILayout.BeginHorizontal ();
        GUILayout.Space (20);
        GUILayout.BeginVertical ();

        foreach (var child in node.children)
            Display (child);

        if (GUILayout.Button ("Add child"))
            node.children.Add (new Node ());

        GUILayout.EndVertical ();
        GUILayout.EndHorizontal ();
    }
}

Instead, you tell Unity not to serialize the tree directly, and you make a seperate field to store the tree in a serialized format, suited for Unity’s serializer:

using UnityEngine;
using System.Collections.Generic;
using System;

public class BehaviourWithTree : MonoBehaviour, ISerializationCallbackReceiver
{
    //node class that is used at runtime
    public class Node
    {
        public string interestingValue = "value";
        public List<Node> children = new List<Node>();
    }

    //node class that we will use for serialization
    [Serializable]
    public struct SerializableNode
    {
        public string interestingValue;
        public int childCount;
        public int indexOfFirstChild;
    }

    //the root of what we use at runtime. not serialized.
    Node root = new Node();

    //the field we give unity to serialize.
    public List<SerializableNode> serializedNodes;

    public void OnBeforeSerialize()
    {
        //unity is about to read the serializedNodes field's contents. lets make sure
        //we write out the correct data into that field "just in time".
        serializedNodes.Clear();
        AddNodeToSerializedNodes(root);
    }

    void AddNodeToSerializedNodes(Node n)
    {
        var serializedNode = new SerializableNode () {
            interestingValue = n.interestingValue,
            childCount = n.children.Count,
            indexOfFirstChild = serializedNodes.Count+1
        };

        serializedNodes.Add (serializedNode);
        foreach (var child in n.children)
            AddNodeToSerializedNodes (child);
    }

    public void OnAfterDeserialize()
    {
        //Unity has just written new data into the serializedNodes field.
        //let's populate our actual runtime data with those new values.

        if (serializedNodes.Count > 0)
            root = ReadNodeFromSerializedNodes (0);
        else
            root = new Node ();
    }

    Node ReadNodeFromSerializedNodes(int index)
    {
        var serializedNode = serializedNodes [index];
        var children = new List<Node> ();
        for(int i=0; i!= serializedNode.childCount; i++)
            children.Add(ReadNodeFromSerializedNodes(serializedNode.indexOfFirstChild + i));

        return new Node() {
            interestingValue = serializedNode.interestingValue,
            children = children
        };
    }

    void OnGUI()
    {
        Display (root);
    }

    void Display(Node node)
    {
        GUILayout.Label ("Value: ");
        node.interestingValue = GUILayout.TextField(node.interestingValue, GUILayout.Width(200));

        GUILayout.BeginHorizontal ();
        GUILayout.Space (20);
        GUILayout.BeginVertical ();

        foreach (var child in node.children)
            Display (child);

        if (GUILayout.Button ("Add child"))
            node.children.Add (new Node ());

        GUILayout.EndVertical ();
        GUILayout.EndHorizontal ();
    }
}

Beware that the serializer, including these callbacks coming from the serializer, usually do not run on the main thread, so you are very limited in what you can do in terms of invoking Unity API. (Serialization happening as part of loading a scene happens on a loading thread. Serialization happening as part of you invoking Instantiate() from script happens on the main thread). You can however do the necessary data transformations do get your data from a non-unity-serializer-friendly format to a unity-serializer-friendly-format.

You made it to the end!

Thanks for reading this far, hope you can put some of this information to good use in your projects.

Bye, Lucas. (@lucasmeijer)

PS: We’ll add all this information to the documentation as well.

[1] I lied, the correct answer isn’t actually 729. This is because in the very very old days before we had this 7 level depth limit, Unity would just endless loop, and then run out of memory if you created a script like the Trouble one I just wrote. Our very first fix for that 5 years ago was to just not serialize fieldtypes that were of the same type as the class itself. Obviously, this was not the most robust fix, as it’s easy to create a cycle using Trouble1->Trouble2->Trouble1->Trouble2 class. So shortly afterwards we actually implemented the 7 level depth limit to catch those cases too. For the point I’m trying to make however it doesn’t matter, what matters is that you realize that if there is a cycle you are in trouble.

Comments (39)

Subscribe to comments
  1. Adam

    August 22, 2014 at 10:35 pm / 

    Seems to be some fairly major bugs in OnAfterDeserialize too.

    But debugging is impossible, because Unity will crash on any attempt to debug that method (since Unity isn’t running on the main thread, you’re not allowed to read any data. If you can’t read, you certainly can’t debug!)

  2. Adam

    August 22, 2014 at 9:16 pm / 

    After spending a couple of weeks trying to make this work, I’ve realised that this feature is – in practice – pretty much useless.

    Unity prevents you from having any “unique ID” per object. Without unique ID’s, it is not possible to do general serialization. Only toy examples will work with these callbacks :(.

    I say “prevents” because Unity seemingly goes out of its way to destroy all stable definitions of ID, and we only need this because the Serialization callbacks above have insufficient information without it. I’m sure this wasn’t intentional, but it’s an inevitable side-effect of three things:

    1. Unity “re-defines” core C# object-construction (constructors aren’t treated as constructors – but the Unity workaround (Awake()) ONLY works at runtime, not in editor), which breaks e.g. Microsoft’s ObjectIdGenerator (try it: Unity serialization kills it)
    2. Unity processes “create new” and “duplicate old” identically: you cannot tell the difference in code. This means you CANNOT generate and save an internal ID: when the object is duplicated, or prefab-instantiated, Unity will give you copies of the same ID, breaking everything)
    3. These callbacks are PREVENTED from comparing GameObject instances. Without that, you cannot construct a workaround based on stable IDs that exist in MonoBehaviour (which – obviously – faced the same problems as above, but has Unity internal fixes to help it along)

    Between these, it’s not possible to write any object graph to be serialized. You can workaround any two of those three (the first two are hardest to workaround: they require you to write code that doesn’t do what you want to do, in order to generate the side-effect you needed).

    But with all 3 … it seems to be hopeless. I’ve tried every avenue I could think of, and always hit the same dead-ends: Unity’s workarounds to Unity’s Serialiazation bugs are incomplete.

    That makes these callbacks IMHO a waste of time. Yes, you can implement some simple, localized data-structures with them. But you can’t have object referneces, which cuts out the vast majority of C# coding :(.

    Very disappointing.

  3. Steam Giveaways

    July 2, 2014 at 2:07 am / 

    Olla buddy, This is really useful! Thank you a lot!

  4. Kelly

    July 1, 2014 at 7:48 am / 

    is the inability to serialize generic types the reason why we can’t have generic monobehaviors on gameobjects? or is there something more to it? Generic monobehaviors would make things a lot easier, rather than having dictionaries of and stub monobehaviors that are just empty implementations of the generic monobehavior.

  5. Lucas Meijer

    June 30, 2014 at 9:44 am / 

    @Ashkan:

    The scenario you mean is when you call Instantiate() on something. Let’s take this example. There are three objects.

    O1: GameObject components=O2, O3
    O2: RigidBody
    O3: BoxCollider

    when you invoke Instatiate(gameObject1), we duplicate all three objects.

    O4: GameObject components=O2, O3
    O5: RigidBody
    O6: BoxCollider

    Notice how the cloned object O4, actually points to O2 and O3 in its component list. this is obviously not what you intended. In the second phase of Instantiate, we fix this up, by running the serializer in a special mode on O1,O2&O3. we ask it “please report your object references”, and then we check if any of the objects referenced were included in the list of objects that were cloned. For each reference that referenced an object that was cloned (both entries in the componentlist in our case), we update that reference to the cloned version instead of the original. after the fix it looks like this:

    O4: GameObject components=O5, O6
    O5: RigidBody
    O6: BoxCollider

    Hope that clears things up, L

  6. Ashkan

    June 30, 2014 at 9:04 am / 

    @Lucas
    Many thanks for writing all of these, It was really helpful but one piece of information was missing. At runtime Unity executes serialization callbacks multiple times, Can you describe the way that it works?
    It should be something like this

    serialize/deserialize (one time)
    if we don’t have any object references exit
    try fixing references to UnityEngine.Object and children (once more)
    //Is there any difference between the times which a fix happens or not?

    Do you serialize structs in the second phase (fixing phase) to avoid reallocating memory for them since they are value types?

  7. Chillersanim

    June 28, 2014 at 10:50 pm / 

    Hello
    This whole serialization sounds great, but…
    When I have a scene with 10’000 objects, all refering to the same prefab and without customized values, then why are they all fully saved?
    My scene save data is always huge, but when I compress this data, it’s only a fraction of the previous size.

    Please add some compression to the saves of scenes, that would be great as people want to share projects with dropbox and stuff.

  8. Lucas Meijer

    June 27, 2014 at 5:48 pm / 

    @Jacob: there are no technical impossibilities to improving the serializer. some improvements are not that hard (generic types), some are a lot of hard work (polymorphism, null). I wrote this post to explain how the serializer currently works, and give some background on some of the reasons why.

    Your serializationcallback is being called all the time because of item#2 in the bulletpoint list of this blogpost: the inspectorwindow serializes the object it is inspecting, and then displays the serialized data.

    @vexe: I don’t see a good usecase for serializing abstract types. I think the only thing it would add is confusion for users who would then (not surprsingly) think that polymorphism will work.

    @vexe: serializing generic types I hope to implement soonish. It turns out it’s a little bit hard because on Metro/WinPhone8 we actually have a completely seperate serialization codepath which is based on codegeneration instead of “setting values in managed memory from c++”. due to the way the serialization code is generated, generic types don’t work quite well with that yet. It will require me changing the way we emit the serialization code for wp8/metro. Internally, we can’t wait until we can throw away the duplicate serialization codepath for WP8/Metro, this is something that will happen when we manage to get wp8/metro shipping on il2cpp

  9. Jacob L

    June 27, 2014 at 1:45 am / 

    @VEXE

    Brilliant solution to the merge issue, thanks! Next thing to solve is how to modify the JSON reference-preserving functionality to use localized ids, so that the reference id’s are only invalidated within the branch that contains graph cycles. *Sigh*

  10. vexe

    June 26, 2014 at 7:27 pm / 

    “But, if i ‘just’ remove the word abstract, the Animal reference persists” – Sorry I take that back. I was testing on a small scale of types. Got confused with something else.

  11. vexe

    June 26, 2014 at 5:43 pm / 

    Another thing on abstract classes: So this is not serialized:

    public class MB : MonoBehaviour
    {
    public Animal animal;
    }

    public abstract class Animal { }

    [Serializable]
    public class Cat : Animal { }

    [Serializable]
    public class Dog : Animal { }

    Assume “animal” is assigned in-editor somehow to a Cat or Dog, via a custom editor or whatever… Of course after I enter playmode the reference is gone.

    But, if i ‘just’ remove the word abstract, the Animal reference persists.

    What ‘practical’ difference is there between the ‘abstract’ Animal and the non-abstract one in this case? – I don’t see any! – There’s still an Animal base referencing child objects. I don’t see why can’t you just serialize abstracts if you could serialize them with the absence of the word ‘abstract’

  12. vexe

    June 26, 2014 at 5:21 pm / 

    “Some things that are relatively long hanging fruit that I want to get to are serialization of generic types”

    I’d love to see that happen.

    @Lucas Meijer: Can you please explain why is it hard to serialize generic types? I mean, you did serialize a generic list, so what’s stopping you from serializing any serializable generic type?

    I’ve seen lots of custom solutions (like Full Inspector for ex) where they use 3rd party serializers like proto-buf to serialize pretty much everything with total ease. If it’s hard to serialize generic types, why can’t you just go for these custom solution/serializers as well?

  13. Imi

    June 26, 2014 at 4:32 pm / 

    @Jacob L
    “…My problem with the as-is callbacks is that my giant JSON string (homegrown polymorphic type support) is printed horrifically in the text-mode assets, making merging a nightmare.”

    We ran into a similar problem with our home-made serialization.

    Just split the (pre-beautified) string on \n and serialize a string[] instead of one string field. Finally, escape ‘ and it looks awesome in the yaml. (I usually do yaml-editing now to do smaller property changes. Much faster.)

  14. Jacob L

    June 26, 2014 at 3:50 pm / 

    @Rene

    Thanks for the post! Unfortunately this whole serialization ordeal is one of my remaining biggest frustrations with Unity. I buy some of the rationales you’ve laid out here, but I strongly suspect that the real reasons have to do with backwards compatibility, not any actual technical impossibility.

    Take polymorphic serialization. The claim of the blog post is that the format of the stream must be known ahead of time. That is, the *format* not the length. Or else you would never be able to serialize a string or a list. Paradigmatically, all you need to support polymorphism is to write a type indicator if the stored type is not the base type. Just the CLR module specification will do, those are unique, yes? In a binary data stream, you of course need a sentinel bit/byte on every class object to indicate whether to check for the type, but you can always make it opt-in on the field level. I needn’t be the one telling you this though, there are freely available serialization libraries that already do this, but Unity’s isn’t one of them.

    The new serialization callbacks would be fully complete if it gave access to the raw serialization stream, as well as text-mode formatting options. Then the users can make as many performance blunders as they wish, and it’s all on their shoulders. My problem with the as-is callbacks is that my giant JSON string (homegrown polymorphic type support) is printed horrifically in the text-mode assets, making merging a nightmare.

    Also, any ideas why OnBeforeSerialize is called constantly on my ScriptableObject when it is open in the inspector?

  15. Ana

    June 26, 2014 at 1:34 pm / 

    “…no longer tear down the entire domain, but to extend mono to re-jit only the methods that have changed when you changed a script”
    +1
    I’d like to see that in the future.
    It makes me happy to see that there are programmers in Unity team that are actively working on improving lives of Unity programmers. These little things, like adding serialization hooks make our lives so much easier and make so many things possible. As a programmer, I would love to see Unity reach a level where I can employ good programming principles and not do workarounds and compromises all the time because some basic things such as polymorphism and generics don’t work well in Unity. I am working with a wonderful language that is C# but can’t use it’s full beauty because of complications. :(

  16. Imi

    June 26, 2014 at 12:29 pm / 

    @Lucas Meijer
    “…no longer tear down the entire domain, but to extend mono to re-jit only the methods that have changed when you changed a script”

    Awesome. IMHO that is the way to go. There are some enterprise-level appdomain serializers out there and for some they were developed as long as C# is around and still don’t get every corner case correctly..

    What about the suggested global hooks in e.g. EditorApplication? I recon, that would be very cheap to realize and support a wide range of applications. You could even pass a parameter whether you do just a hot-fix to the runtime or you are going for a full assembly reload.

  17. Joachim

    June 26, 2014 at 12:12 pm / 

    @Prankard:
    Types like GameObject, Components, MonoBehaviours (Anything inheriting from UnityEngine.Object) are simply referenced, no duplication etc is going on.

    Other types like structs and classes with [system.serializable] not inheriting from UnityEngine.Object are “embedded” into the serialization stream and are thus duplicated.

  18. Prankard

    June 26, 2014 at 11:40 am / 

    Hmm, does that mean if I did this:

    // Assigned in scene
    public GameObject prefab;

    void Start()
    {
    GameObject instance = (GameObject)GameObject.Instantiate(prefab);
    }

    The scene is loading in my serialised object into a new GameObject on scene load. And then on Instantiate it is serialising that new GameObject and then deserialise and making a new one?

    Is there not a way for us to Instantiate a prefab as a new GameObject directly from the serialised prefab instance? As this would be faster if my Instantiate was in a large for loop.

  19. Prankard

    June 26, 2014 at 11:00 am / 

    This was an interesting blog post.
    It makes me understand a little bit more on the core code of Unity3D. Thanks for sharing :)

    I had a game that I made (back in Unity version 3) that had a particular ‘levelblock’ prefab.
    This prefab was a reskinable 3d block that we used to build a lot of custom levels out of.
    One of the things we realised (pretty quickly), was that if you wanted to Instantiate() about 200-300 prefabs it makes mobile run pretty slowly for a few seconds.

    After reading your post it seems clear why, (creating from a serialised prefab and then doing all those reference checks after.
    Makes me think that in those rare scenarios you may not want to use a unity prefab and perhaps try and create that one game object with all new keywords and reference the dependencies yourself.

    The serialiser you’ve written seems interesting. Is there and api at runtime that we could call it to serialise anything that extends Unity.Object and then deserialise it ourselves? It would be a good alternative to c#’s serialiser. (Or does it already use the system.serializer as its core?)

  20. Lucas Meijer

    June 26, 2014 at 8:38 am / 

    @Hatless: I agree that the practicle usecases for having a running game survive a serialize-domainreload-deserialize phase are tiny. I think that if we would also serialize static variables it would go from tiny to a little more than tiny. What we’ve been thinking about internally is to completely change the way around that we do script reloads, and to no longer tear down the entire domain, but to extend mono to re-jit only the methods that have changed when you changed a script. (and do a traditional domain reload in case you moved/added/changed field layout). speeding up entering playmode is defenitely something that has been discussed a lot around the dev team lately.

    Regarding communication of this feature on the website: I just spent five minutes looking to see if anywhere we advertise that you can change scripts while keeping the game running. I was not able to find it. If you can point me to a piece of text on the website that you feel is advertising this feature as if it does more than it does in reality, please point me to it, and I’ll change the website.

  21. Lucas Meijer

    June 26, 2014 at 8:33 am / 

    @richard: if you have a monobehaviour with 3 int fields. create some of them in the scene, and populate the scene with data. if you later change your script, and add another int field in the middle of the other fields, the typetree is the thing that will automatically “upgrade your data” to the new format, without you even having to think about it. we use typetrees for this for monobehaviours, but even for our internal types like texture2d or camera, it’s very easy for us to add another member variable that needs to be serialized. the typetree will automatically make sure we can properly load old data. you could totally design a serialization system to work differenly, have different tradeoffs. ours is currently designed like this. I’d like to improve the serializer over time, (as we did with unity4.5), but we have to weigh each potential improvement against all other unity improvements that we could be doing. Some things that are relatively long hanging fruit that I want to get to are serialization of generic types, a built in dictionary (like the one you and I started hacking on a bit back, but then built on serializationcallbacks).

    @richard: about cycles in private members, I think the answer to that is yes. but I didn’t check. if the answer indeed is yes though, you can easily break it by adding a [NotSerialized] attribute

  22. Richard Fine

    June 26, 2014 at 2:27 am / 

    Also, regarding private members: Doesn’t them being serialised for hot-reload mean we can’t have type cycles even in private members?

  23. Richard Fine

    June 26, 2014 at 2:25 am / 

    So I get the typetree vs instance distinction, but I don’t really get *why* it’s that way.

    Given that you can have a collection of variable length in an instance of a type, you can’t use the type tree to predict the size of an instance, or even the offsets of particular fields within an instance.

    So I’m confused, what’s the typetree actually *for*?

  24. Imi

    June 26, 2014 at 1:03 am / 

    I second Hatless. “This core feature of Unity — this feature you advertise very prominently on the website — has not, so far as I know, ever worked.”

    Every Unity programmer I ever talked to confirmed me, that he never saw a Unity project bigger than a toy-solution or tutorial, that actually could survive an assembly reload during play mode.

    For our project, we wrote a small editor script that immediately stops play mode when it detects an recompile. This is much faster than to wait for the assembly reload to occur just to throw tons of exceptions. Also, we mark all and every private variable as NonSerializable to speed up assembly reload – even if its just a bit.

    I know that it is possible to write code that survive assembly reloads, but it restricts so much of sound C# design principles, that it is laughable to even try.

    Please do not assume that private member serialization during runtime is anything usefull in its current state. Some kind of hot-swapping of dlls during runtime without unloading the appdomain would be awesome. (Like MS.NET and Visual Studio does it for some code changes)

    An global callback in EditorApplication just before and after assembly reload occurs would help much more (which is guaranteed to be in the main thread, so it actually can do usefull things with Unity). This way we could fire up some professional C# appdomain serializer (or redirect to any ingame savegame/loading system.. nice integration test for these as well ^^).

  25. hatless

    June 25, 2014 at 4:41 pm / 

    Will Unity ever stop losing static values when it recompiles scripts at runtime?

    This (invisibly, silently) breaks any project over a certain level of complexity.

    This core feature of Unity — this feature you advertise very prominently on the website — has not, so far as I know, ever worked.

  26. Lucas Meijer

    June 25, 2014 at 2:44 pm / 

    @all: what rene said.

    it’s not technically impossible to ever support null. it’s a lot of non trivial work though. We’ have to somehow serialized “inline objects” with a bool wether or not this one is null. it affects how you interact with such objects with the SerializerProperty class, as well as the prefab system. (if the “isnull” bit is marked, but a prefab sets a value anyway, what do you do). none of these are theoretically unsolvable. you would however still run into the depth limit problem, because of the way we do backwards compatible loading. When we do backwards compatible loading, we at runtime, generate a typetree for a certain object. this concept of a typetree is actually a pretty core one in unity, and already should give a good feeling on how many of our systems are built around assumptions on how datalayout is static. we indeed have the concept of a collection, but other than that that’s it. so when we generate a typetree, we actually create an object, then we serialize it in a special typetree creation mode. if you have class cycles, the typetree would still grow very big. (we cap it to 7 levels too).

    so yeah, a ton of work. up until now we have prioritized other things, and I don’t see that changing in the near future. (I actually spent a week or two going down this rabbit hole for both null and polymorphism when I did the serialization improvements for 4.5, thinking I’d be able to get something in, but I ended up with the conclusion that it would take a lot more time than that, and that my time was better spent providing things like serialization callbacks, and other things in Unity that I feel could really use some loving).

  27. Benoit Fouletier

    June 25, 2014 at 2:43 pm / 

    Thanks for the post Lucas!
    Regarding this: “No support for null for custom classes”
    This has its pros and cons, but at least it’s a very clear contract… except when the script just got created, where custom class fields (and arrays, BTW) are null until the first reload or inspector change.
    I reported this, case # 613469, just in case it didn’t make its way to you ;) !
    This makes it painful for custom editor and ExecuteInEditMode scripts, this forces us to place nullchecks for just this corner case.

    This is sort-of related (at least in my mind) to another bug (case # 608574): OnValidate is not called when the script is first created. It’s like “just created” objects don’t go through the same codepath that “properly serialized and loaded”.

  28. René Damm

    June 25, 2014 at 2:04 pm / 

    @Richard
    This has to do with the separation between “this is how you serialize class Node” vs. “serializing an instance of class Node”. Basically, looking at any arbitrary class C, the system has to be able to figure out what the data for the class looks like on disk and that has to be true for any given instance of class C.

    So, let’s say the system is looking at the Node class that has the recursion. It finds the List field. So the system switches to the List serialization code which says “I’m inserting a count and an arbitrary number of Node objects at the current position”. That in turn makes the system switch back to the serialization code for Node which says “I’m inserting a Node at the current position”. And round it goes.

    The serialized data that you outline, however, depends on how things actually look at runtime, i.e. on the actual values found in properties, whereas the serialization system operates entirely from static type data.

    And you’re correct about private fields being serialized when we do script reloads in the editor. The rationale here is that if we don’t, we lose significant internal state (both for the editor as well as when your game is running) and when resuming execution things will likely go very wrong. However, this serialization is completely transient, i.e. it only ever writes state to memory and never to disk.

  29. René Damm

    June 25, 2014 at 1:36 pm / 

    @Daniel
    It’s because the system as is has no concept of “pointer/reference to something that is not a UnityEngine.Object”. The reason for that is that in the object persistence model (which is based on C++ objects, not .NET objects), these objects that aren’t UnityEngine.Objects have no identity. A UnityEngine.Object always has an identity established with the engine (which you can see by calling Object.GetInstanceID) and every such object can be made persistent in which case it receives another identity that is tied to a blob of data on disk.

    So, for correctly handling null correctly on arbitrary user classes, the persistence system would have to have a concept of “pointer to a shared piece of data in a serialized data stream which, however, is not an engine object”. With that, it would be possible to serialize reference as you’d expect rather than treating custom user classes as value types.

    However, there’s other ways to hack in support for a “null” representation without adding full support for a new type of pointer/reference. We’ll have to see what’s the best path here in the future.

  30. John

    June 25, 2014 at 11:57 am / 

    @RICHARD FINE
    I’m curious about the zero-length lists too. Will it still run into the problem if a list down the line is empty before the depth limit is reached?

  31. Ana

    June 25, 2014 at 11:23 am / 

    Thank you for finally writing this post. Now I finally understand why I had so much trouble with serialization in the past.
    You should add articles like these to the documentation. They are very valuable!

    Serialization callbacks, finally! Woooohooo!!! Oh how long I’ve waited for those!
    And this time it is not a magical callback but interface implementation!!
    And to anyone who wonders if this works with custom non MonoBehaviour classes – it does! (But not structs)
    You have no idea just how happy you just made me!

  32. Dmitriy Pyalov

    June 25, 2014 at 11:23 am / 

    Is there something special about ScriptableObjects, as they seem to serialize its private fields by default?

  33. Chillersanim

    June 25, 2014 at 9:39 am / 

    I’m also curious about the behaviour that Richard Fine posted.

    But in addition, why don’t you support polymoorphism?
    Wouldn’t it be easy to just get the type (object.getType()) and store this type together with the data? I don’t think that this would cause a large overhead, but at the end the serialization process would work like the developer would think it does.

    And finaly, why don’t you support properties that are writable and readable? It should be easy with reflection…

    Greetings
    Chillersanim

  34. Beck Sebenius

    June 25, 2014 at 3:55 am / 

    I’m curious about the same thing Richard Fine posted, I read your example of that problem in a Unity thread and had some confusion as to why that would be a problem.
    Thanks for posting this writeup, though. The serialization callbacks are an exciting advancement for editor scripting – I can’t wait to upgrade to 4.5.

  35. Lior Tal

    June 25, 2014 at 12:35 am / 

    I wonder if other engines also put such strong emphasis on the concept of serialization or it “just works” as the developer would expect (e.g: UE4)

  36. Richard Fine

    June 24, 2014 at 10:14 pm / 

    While you don’t support nulls, you do support serialising zero-length lists of objects. So how come [Serializable] class Node { public List ns; } causes problems, given that ns can be zero-length? I mean, how come it can’t generate a small set of properties like

    ns.arraySize = 3
    ns.data[0].ns.arraySize = 0
    ns.data[1].ns.arraySize = 1
    ns.data[1].ns.data[0].arraySize = 0
    ns.data[2].ns.arraySize = 0

    and stop there?

    Also, you said that a field has to be public (or [SerializeField]) to be serialised, but Levi said in the forum that private fields get serialised during hot-reload in the editor…?

  37. fatima

    June 24, 2014 at 9:33 pm / 

    desi

  38. Daniel

    June 24, 2014 at 8:21 pm / 

    Why don’t you just support nulls? I have my own serialization system and don’t use scenes and I didn’t encounter any issues for this for my use-cases; what I serialize is a little more verbose though. Is this for performance reasons, I’d like to know in-case it’s something that hasn’t yet occurred to me.

  39. Brian Turner

    June 24, 2014 at 6:39 pm / 

    Nice, I’ve actually gone in and edited yaml in a text editor when opening unity would incur an import time longer than the time need for the change. I’ve also done it to batch edit settings on AudioClips.

Comments are closed.