The new Asset Import Pipeline: Solid foundation for speeding up asset imports
As of 2019.3, the new Asset Import Pipeline is the default for new projects, aiming to save you time with faster platform switching and laying the foundation for faster imports. We’re also making the asset pipeline scale better for very large projects. Read on to find out more about the new improvements, and our motivation and considerations.
Whenever you put a new asset into your project, it doesn’t actually become a part of the project until the Asset Import Pipeline discovers and imports it. The correct detection of the project state is what the Asset Import Pipeline is responsible for while allowing you to query for this state through various APIs.
In 2017, the Asset Import Pipeline rewrite work began to pave the way towards a more robust and scalable approach, while also addressing a number of pain points reported by you in your daily workflows. With Unity 2019.3 (now available in beta), the new Asset Import Pipeline (also known as Asset Import Pipeline V2) will be the default implementation for new projects. Older projects can choose to upgrade to the new Asset Import Pipeline to get the benefits of this new system.
Now is a good time to share some of the thinking behind the new pipeline. Specifically, we want to share the considerations taken to make sure the new system is compatible with the existing APIs so that scripts will not have to be re-written when upgrading to the new Asset Import Pipeline.
Achieving faster import times
There are numerous workflows that are part of a daily development cycle. We have identified the most time-consuming issues and have implemented solutions for them.
Importing assets can take a long time. Converting the source data to a format that the Unity Editor, or a platform, is ready to utilize is not a trivial process. For example, importing a complex 3D model requires a large number of computations, and when combined with animation this time can quickly grow.
To address this there are 3 key concepts which need to be addressed as part of the solution:
With most types of assets, Unity needs to convert the data from the source file, depending on the target platform for your projects. The result will vary between compatible GPU formats such as PVRTC, ASTC or ETC.
This is because most file formats are optimized to save storage, whereas in a game or any other real-time application, the asset data needs to be in a format ready for immediate use by hardware, such as the CPU, graphics, or audio hardware.
For example, when Unity imports a PNG image file as a texture, it doesn’t use the original PNG-formatted data at runtime. Instead, when the texture is imported, Unity creates a new representation of the image in a different format which is stored in the project’s Library folder. This imported version is what is used by the Texture class in the engine, and uploaded to the GPU for real-time display. This is referred to as the Import Result.
We need to know that both you and I get the same Import Results in the same exact format when we import, even when we’re using different hardware. The principle of getting the same output for a given input is what we call determinism.
The Asset Import Pipeline keeps track of all the dependencies for each asset and keeps a cache of the imported versions of all the assets. An asset’s import dependencies are all the data that could influence the import result. This means that if any of your asset’s import dependencies are changed, the cached version of the imported asset becomes outdated, and the asset needs to be re-imported to reflect those changes.
Introducing the new Asset Import Pipeline
There are different situations where importing can take a long time. We’ve identified the following two workflows and implemented two solutions to address the above issues: Fresh Project Import and Fast Platform Switching.
Fresh Project Import
When you set up a project for the first time, it’s essentially the same as when the Library folder is deleted. This means that every asset in the assets folder needs to be enumerated and imported by the Asset Import Pipeline. This is naturally an expensive operation. However, by ensuring that our import process is deterministic and stable across machines, the time it takes to retrieve import results can be reduced by many orders of magnitude, depending on the size of the Source Asset and the size of the Import Result.
We achieve this by using the new Unity Accelerator which caches import results on the cloud from anyone who is connected to it, thus allowing you to directly download the import results from a server rather than having to go through the heavy processing which importing an asset would entail.
Fast Platform Switching
Up until Unity 2019.2 (with the original Asset Import Pipeline), the Library folder was comprised of the GUIDs of Assets being their filename. Thus, switching from a platform to another platform would invalidate the Import Result in the Library folder, causing it to be re-imported every time you switch platforms.
If you had to switch back and forth between platforms multiple times per day, this could easily take up hours, depending on your project size.
Some of you have figured out workarounds for this, such as having a copy of your project per platform on your machine, but that doesn’t scale very well.
With the new Asset Import Pipeline, we’ve removed the GUID to File Name mapping. Since dependencies for a particular Asset are tracked, we are able to Hash them all together to create a revision for the Import Result of an Asset. This allows us to have multiple revisions per Asset, which then means that we are no longer bound by a GUID to File Name mapping. Not having this requirement allows us to have Import Results which work across different configurations. For Fast Platform Switching, we could have an Import Result per platform, so that when you switch platforms back and forth the Import Result is already there, thus making the platform switch many orders of magnitude faster than with the Asset Import Pipeline V1.
As you make changes to assets, Unity generates a number of new files that will get generated. This will take up more storage space on your disk. However, the way we have decided to approach this issue is to remove unused Import Results when Unity restarts. We keep track of the latest import result per platform so that Fast Platform Switching can still take place while older Import Results are removed, thus helping you free up some of your disk space.
How to upgrade to the new Asset Import Pipeline
The new Asset Import Pipeline is available with Unity 2019.3 beta. If you have an existing project, you can upgrade to the new Asset Import Pipeline using the Project Settings Window in the Editor:
Selecting Version 2 will tell the editor you now want to use the new Asset Import Pipeline together with this project, and restarting your project will re-import it using the new Asset Import Pipeline code. This essentially has the same effect as deleting your Library folder, but without deleting it. When switching to use Asset Import Pipeline V2, the Import Results from the original Asset Import Pipeline are not deleted as V2 creates its own folder structure to store its Import Results.
Projects that you’ve created in Unity 2019.2 or older will still use the original Asset Import Pipeline by default. When opening such a project in Unity 2019.3 for the first time, you’ll get an option to upgrade to the new Asset Import Pipeline. If you decline, your project will continue using the original Asset Import Pipeline. Furthermore, the selected version is stored in the EditorSettings.asset file of your project, so it can be version controlled.
New Projects created with the new Asset Import Pipeline
When creating a new Project with Unity 2019.3 or newer, the new Asset Import Pipeline has become the new default way to work. All new projects you create will be using it.
More improvements coming
At Unite Copenhagen 2019, our team presented two talks. My talk is a general introduction to the topics covered in this blog post and can guide your decision-making for your own Asset Management strategies. My colleague Jonas Drewsen talked about the upcoming features directed towards making the asset pipeline more extensible and ensuring project stability:
Now available in Unity 2019.3 beta
Get the Unity 2019.3 beta and try out the new Asset Import Pipeline. We’re looking forward to hearing what you think on the forum! You can also get in touch with me on Twitter if you have further questions.