Categories & Tags
Archive

In-depth Cache Server

October 26, 2012 in Technology, Unity Products and Services by

In Unity 3.5 we shipped the first version of the Cache Server, and with 4.0 we’re making it even better (the way it works is the same). I want to go a bit in-depth to explain what it actually does, because it’s pretty awesome and can save you lot of time.

It makes importing projects insanely fast.

If you pull changes from version control and everyone in your team uses the cache server, everything will already be cached and your imports will be almost instantaneous.

If you switch between platforms, you no longer need to wait for a long time while Unity re-compresses textures. Switching platforms becomes almost instant.

It works with any version control system: SVN, Perforce, Asset Server, Git, Hg etc.; or if you don’t use version control. The only requirement is that you enable .meta files in the project.

The Cache Server helps you make this go away:

 

Version control in Unity

In Unity all source assets are stored in the “Assets” and “ProjectSettings” folders. Since these two folders contains all source assets, they must be in version control and they are the only folders that need to be in version control. The “Library” folder should not be under version control. In order to use Version Control in Unity you also need to enable .meta files. This can be done in the Edit -> Project Settings -> Editor menu.

The Cache Server is not a replacement for version control. In fact it does not touch source assets at all! It is a way of caching the import step. Thus it works with any version control system out there, simply because it does not interfere with it in any way.

 

What does the import step do?

Unity has a completely automatic asset pipeline. Whenever a source asset like a .psd or .fbx file is modified, Unity automatically reimports it. During the import step Unity creates a file with the imported data. Various things are generated in this import step, for example a .psd file will import the texture and compress it for the target platform. A Texture2D class object will be generated and stored in the “Library/metadata” folder. Unity also generates a preview for the texture which is used when picking assets. The generated data is not under version control, since it can always be regenerated. Some asset types will reimport when switching platforms, for example textures require a reimport when switching to a platform with different compression scheme. Meshes on the other hand are independent of the target platform.

The import step sometimes does very complex operations. For example when importing a .psd file for iOS we generate PVR compressed texture automatically. Or we could generate secondary mesh UV data for lightmapping. In some cases it can take seconds for each asset. This time can add up and thus the cache server can give you massive import speed benefits.

In Unity each asset is identified by a GUID (a globally unique identifier). The GUID is generated when an asset is first imported and stored in the .meta file. We do this so we can move any asset without losing any references to it (all asset references are GUID-based, not folder or file name based).

 

What does the Cache Server do?

The cache server accelerates the import step. It does not touch source data. The cache server is independent of version control. It is even independent of your current project.

Instead of actually importing an asset Unity will re-use an imported asset another team member or yourself has previously imported and uploaded to the cache server. The Cache Server is completely automatic and happens as part of the import pipeline. How much benefit you get from the cache server depends on the type of assets. Textures and large model files usually take the most time to import thus the speed up will be greatest there. Anything from 2x-100x can be expected!

The first thing Unity does when it is about to import an asset is to generate an MD5 hash of all source data.

For a texture this consists of:

  • The source asset: “myTexture.psd” file.
  • The meta file: “myTexture.psd.meta” (stores all importer settings).
  • The internal version number of the texture importer.
  • A hash of all version numbers of all Asset Postprocessors.

Anything that can affect the resulting imported data will be hashed. The Cache Server is a very simple file server that looks up assets by the GUID & Hash of that asset. On the cache server you might have many cached files for the same texture. The cache server removes assets that have not been used for a period of time automatically. It uses a simple least recently used scheme for throwing away unused caches.

If you switch Unity from iOS to Standalone mode for example, the imported texture will get a different hash. When switching the platform Unity calculates the new hash of all textures and tries to download them from the cache server. If they are available they will be used, otherwise it will import the texture and upload the imported result to the cache server.

Once it is cached on the server it can be used next time someone switches to the platform or imports that asset. Thus switching platforms becomes pretty much instant.

 

Cache Server performance

The Cache Server itself is very simple. It is essentially a streaming file server. If you setup a cache server for a big team, the important considerations are:

  • Lots of RAM so the cache server can keep all assets in disk cache and serve them instantly
  • Fast disk so if it’s not in RAM cache it can be served quickly
  • Ensure the cache server is at least 3-4x larger than the Library/metadata folder in your project
  • The network connection to the machines using Unity must be fast.

Since it is purely a cache that accelerates the import and all the data can always be re-generated automatically by reimporting, it usually makes sense to ensure the cache server is local. Some people setup cache servers that are used remotely, this can make sense with a very fast internet connection. It is best to profile it and see what is fastest. But for the best performance I’d recommend  to have the cache server in the same location with a wired Ethernet connection. There is no problem with having multiple cache servers for one project. It is simply a cache, there are no conflicts if you have multiple servers.

Once the data is downloaded, the actual integration of the data usually takes less than 1 ms.

Essentially it caches the entire import step including preview generation, meaning other people using the same cache server can reuse the already imported asset without having to go through the full import process. In Unity the asset importers often perform complex tasks for example, when importing a .psd file in iOS we generate compressed PVRTC textures automatically for you so you get the best performance on the device. Or we have an option to generate secondary UV data for your models for lightmapping purposes automatically. These tasks can easily take several seconds and when you have a ton of assets it can quickly add up.

 

Cache Server on Windows

The Cache Server is built on top of node.js. The windows file system & file locking behaviour on windows make the cache server work not very efficiently on windows. I suggest that you install your cache server on a Linux or OS X server instead.

What not to do in the Cache Server?

The Cache Server does not handle asset dependencies. In fact, Unity’s asset pipeline does not deal with the concept of asset dependencies either; it is built in a way to avoid dependencies. AssetPostprocessors are a common technique used to customize the asset importer to fit your needs; for example you might want to add MeshColliders to some GameObjects in an FBX file based on their name or tag.

However, it is also easy to use AssetPostprocessors to introduce dependencies. For example you might use data from a text file next to the asset to add additional components to the imported game objects. This is not supported in the Cache Server! If you want to use the cache server you you will have to remove dependency on other assets in the project folder. Since the Cache Server doesn’t know anything about the dependency in your postprocessor, it will not know that anything has changed thus use an old cached version of the asset.

In practice there are plenty of ways how you can do asset postprocessing that works well with the Cache Server.
You can use:

  • The Path of the imported asset
  • Any import settings of the asset (including string AssetImporter.userData)
  • The source asset itself or any data generated from it passed to you in the asset postprocessor.

Modifying materials that already exist might cause trouble as well. When using the cache server Unity validates that the references to materials are maintained. But since no postprocessing calls will be invoked, the contents of the material can not be changed when a model is imported through the cache server. Thus you might get different results when importing with or without cache server. It is best to never modify materials that already exist on disk from an asset postprocessor.

The issue with modifying materials that already exist is that, if you download an fbx file through the cache server, then there is no import process running for it, this means that if you rely on resetting the generated materials to some generated defaults every single time the model importer runs, then this asset postprocessor will not be run, when importing a cached fbx file.

 

Where do I get it?

The Cache Server is part of the Team License: http://unity3d.com/unity/team/

You can download it here: http://unity3d.com/unity/team/cacheserver/

How to setup the Cache Server: http://docs.unity3d.com/Documentation/Manual/AssetCacheServer.html

Conclusion

The Cache Server gives you massive import time speed ups. If you work on a large project folder, using it is incredibly important. It makes your life better, seriously. Use it!

Share this post

Comments (13)

Comments are closed.

r61
26 Oct 2012, 7:50 am

is there any specific reason why aren’t .meta files enabled by default since they are quite useful/even necessary for any version control system to use with Unity ?

Joachim Ante
26 Oct 2012, 8:17 am

Some people don’t like that they get all those extra files in the project. But I tend to agree. That would be a good change.

26 Oct 2012, 10:31 am

Very nice writeup on the Cache Server, will hopefully shed a lot of light on the matter as there seems to be some pretty big confusion or missassumptions floating around on it (or just an absolut lack of knowledge that it even exists)

Foriero
26 Oct 2012, 10:36 am

Please sell Cache Server separately. For 99$. And make it as Unity Local Cache solution and Unity Server Cache solution. That means for teams one would use Server solution and for individual needs one would use Local Solution. Or please add to Pro version this local cache solution, it is unbearable to switch platforms, I always get angry … :-) For 500$ it is too much for local solution given that there are much better svn or perforce clients than you have. I would love to pay 99$ for local cache solution if it would be available.

Mr. B
26 Oct 2012, 7:47 pm

Awesome writeup, thanks for this.

Please don’t turn on meta files be default. Instead, find a way to get rid of them altogether please.

Chris
27 Oct 2012, 4:19 am

+1 for the $99 separate cache server for solo users.

27 Oct 2012, 10:13 am

Always enjoy learning more about Unity’s advanced functionality directly from the experts that built it!

Though it has already been mentioned, I too would be thrilled with a local caching system included in the Pro license to accelerate platform switching.

laurent
28 Oct 2012, 5:10 pm

“It is best to never modify materials that already exist on disk from an asset postprocessor.”

It sounds like you are saying don’t change materials generated by import (the import containing posprocessor)
Can you explain with an example.

Joachim Ante
29 Oct 2012, 3:31 am

@Laurent: I updated the post to have some more info on postprocessors. Let me know if that makes it clearer.

29 Oct 2012, 4:07 am

+2 for solo license

29 Oct 2012, 4:31 pm

Importing is one important process. If this can really speed things up, I’ll get one. Hopefully, the setup wont be to complicated.

DynamicHead
7 Nov 2012, 5:14 am

Thank you for explaining.
We really love the idea to open projects without delay.

28 Dec 2012, 4:51 am

My brother recommended I might like this blog. He was once totally right. This submit actually made my day. You cann’t believe just how so much time I had spent for this information! Thanks!

Leave a Reply

Comments are closed.