Search Unity

The new Asset Import Pipeline: Solid foundation for speeding up asset imports

, 11月 7, 2019

As of 2019.3, the new Asset Import Pipeline is the default for new projects, aiming to save you time with faster platform switching and laying the foundation for faster imports. We’re also making the asset pipeline scale better for very large projects. Read on to find out more about the new improvements, and our motivation and considerations.

Whenever you put a new asset into your project, it doesn’t actually become a part of the project until the Asset Import Pipeline discovers and imports it. The correct detection of the project state is what the Asset Import Pipeline is responsible for while allowing you to query for this state through various APIs.

In 2017, the Asset Import Pipeline rewrite work began to pave the way towards a more robust and scalable approach, while also addressing a number of pain points reported by you in your daily workflows. With Unity 2019.3 (now available in beta), the new Asset Import Pipeline (also known as Asset Import Pipeline V2) will be the default implementation for new projects. Older projects can choose to upgrade to the new Asset Import Pipeline to get the benefits of this new system.

Now is a good time to share some of the thinking behind the new pipeline. Specifically, we want to share the considerations taken to make sure the new system is compatible with the existing APIs so that scripts will not have to be re-written when upgrading to the new Asset Import Pipeline.

Achieving faster import times

There are numerous workflows that are part of a daily development cycle. We have identified the most time-consuming issues and have implemented solutions for them.

Importing assets can take a long time. Converting the source data to a format that the Unity Editor, or a platform, is ready to utilize is not a trivial process. For example, importing a complex 3D model requires a large number of computations, and when combined with animation this time can quickly grow.

To address this there are 3 key concepts which need to be addressed as part of the solution:  

Import Result

With most types of assets, Unity needs to convert the data from the source file, depending on the target platform for your projects. The result will vary between compatible GPU formats such as PVRTC, ASTC or ETC. 

This is because most file formats are optimized to save storage, whereas in a game or any other real-time application, the asset data needs to be in a format ready for immediate use by hardware, such as the CPU, graphics, or audio hardware.

For example, when Unity imports a PNG image file as a texture, it doesn’t use the original PNG-formatted data at runtime. Instead, when the texture is imported, Unity creates a new representation of the image in a different format which is stored in the project’s Library folder. This imported version is what is used by the Texture class in the engine, and uploaded to the GPU for real-time display. This is referred to as the Import Result.

Determinism

We need to know that both you and I get the same Import Results in the same exact format when we import, even when we’re using different hardware. The principle of getting the same output for a given input is what we call determinism. 

Dependency Tracking

The Asset Import Pipeline keeps track of all the dependencies for each asset and keeps a cache of the imported versions of all the assets. An asset’s import dependencies are all the data that could influence the import result. This means that if any of your asset’s import dependencies are changed, the cached version of the imported asset becomes outdated, and the asset needs to be re-imported to reflect those changes.

Introducing the new Asset Import Pipeline

There are different situations where importing can take a long time. We’ve identified the following two workflows and implemented two solutions to address the above issues: Fresh Project Import and Fast Platform Switching.

Fresh Project Import

When you set up a project for the first time, it’s essentially the same as when the Library folder is deleted. This means that every asset in the assets folder needs to be enumerated and imported by the Asset Import Pipeline. This is naturally an expensive operation. However, by ensuring that our import process is deterministic and stable across machines, the time it takes to retrieve import results can be reduced by many orders of magnitude, depending on the size of the Source Asset and the size of the Import Result.

We achieve this by using the new Unity Accelerator which caches import results on the cloud from anyone who is connected to it, thus allowing you to directly download the import results from a server rather than having to go through the heavy processing which importing an asset would entail.

Fast Platform Switching

Up until Unity 2019.2 (with the original Asset Import Pipeline), the Library folder was comprised of the GUIDs of Assets being their filename. Thus, switching from a platform to another platform would invalidate the Import Result in the Library folder, causing it to be re-imported every time you switch platforms.

Asset Import Pipeline V1

If you had to switch back and forth between platforms multiple times per day, this could easily take up hours, depending on your project size.

Some of you have figured out workarounds for this, such as having a copy of your project per platform on your machine, but that doesn’t scale very well.

With the new Asset Import Pipeline, we’ve removed the GUID to File Name mapping. Since dependencies for a particular Asset are tracked, we are able to Hash them all together to create a revision for the Import Result of an Asset. This allows us to have multiple revisions per Asset, which then means that we are no longer bound by a GUID to File Name mapping. Not having this requirement allows us to have Import Results which work across different configurations. For Fast Platform Switching, we could have an Import Result per platform, so that when you switch platforms back and forth the Import Result is already there, thus making the platform switch many orders of magnitude faster than with the Asset Import Pipeline V1.

Asset Import Pipeline V2

Possible downside

As you make changes to assets, Unity generates a number of new files that will get generated. This will take up more storage space on your disk. However, the way we have decided to approach this issue is to remove unused Import Results when Unity restarts. We keep track of the latest import result per platform so that Fast Platform Switching can still take place while older Import Results are removed, thus helping you free up some of your disk space.

How to upgrade to the new Asset Import Pipeline

The new Asset Import Pipeline is available with Unity 2019.3 beta. If you have an existing project, you can upgrade to the new Asset Import Pipeline using the Project Settings Window in the Editor:


Selecting Version 2 will tell the editor you now want to use the new Asset Import Pipeline together with this project, and restarting your project will re-import it using the new Asset Import Pipeline code. This essentially has the same effect as deleting your Library folder, but without deleting it. When switching to use Asset Import Pipeline V2, the Import Results from the original Asset Import Pipeline are not deleted as V2 creates its own folder structure to store its Import Results.

Projects that you’ve created in Unity 2019.2 or older will still use the original Asset Import Pipeline by default. When opening such a project in Unity 2019.3 for the first time, you’ll get an option to upgrade to the new Asset Import Pipeline. If you decline, your project will continue using the original Asset Import Pipeline. Furthermore, the selected version is stored in the EditorSettings.asset file of your project, so it can be version controlled.

New Projects created with the new Asset Import Pipeline

When creating a new Project with Unity 2019.3 or newer, the new Asset Import Pipeline has become the new default way to work. All new projects you create will be using it.

More improvements coming

At Unite Copenhagen 2019, our team presented two talks. My talk is a general introduction to the topics covered in this blog post and can guide your decision-making for your own Asset Management strategies. My colleague Jonas Drewsen talked about the upcoming features directed towards making the asset pipeline more extensible and ensuring project stability:

Now available in Unity 2019.3 beta

Get the Unity 2019.3 beta and try out the new Asset Import Pipeline. We’re looking forward to hearing what you think on the forum! You can also get in touch with me on Twitter if you have further questions.

15 コメント

コメントの配信登録

返信する

これらの HTML タグや属性を使用できます: <a href=""> <b> <code> <pre>

  1. Thanks for the work and blog-post on the import pipeline!

    I have a question about “remove unused Import Results when Unity restarts”. I don’t fully understand this. When becomes an import result unused? Is this related to deleting assets from the Assets directory and the import results weren’t deleted in V1? Or is there some clever logic that even detects “this target platform specific import result wasn’t used for 28 days, I need to delete this now!”?

    1. Hi Peter,
      An import result becomes unused when it is not the latest revision.
      For example, when you change either the contents of a texture or its dependencies, a new import result will be generated on disk and this is what we store as the latest revision of that Asset’s import result.
      When switching platforms for the first time, we naturally have to re-import the project since the Platform is in itself an import dependency.
      When switching back to the previous platform you were on, we check if the import result is there and prevent an import from happening, thus allowing the fast platform switch to happen.
      The, what we do when restarting the editor is that if we detect an import result doesn’t match what we have stored in the latest revision, then we remove it. There is one exception to this, where if we detect that the platform doesn’t match but all the other dependencies match, then we don’t remove the import result from disk.
      All this work happens to files inside the Library folder, so your source assets (i.e. everything under the Assets folder) stays the same.

  2. Cool. When will you support devs with imperfect vision and retina display users?

  3. I really appreciate you’re trying to optimize loading speeds. Workflow speed has always been a big selling point for me. I have been a bit worried about some design-decisions last years, such as including too many packages per default on new projects, increasing loading times and complexity, and cluttering my assets view with packages. I prefer the possibility to start from a clean state, where I’m in total control, which used to be the case.

    Do you see any possibility of Unity reducing project file size and file count in the future? I’ve always been a bit baffled by how a simple project with a few textures and models suddenly take up 3 Gb and include 30 000 files, making it time-consuming to backup in e.g. Dropbox.

    It seems like textures occupy huge amounts of space, which I can understand if they’re re-generated as a less compressed format. However, I don’t really understand how the file count can grow so high, even if I count one extra meta file per asset.

    Do you think it would be possible for Unity to store textures in a format that take up less space, at least during development? And find another way to save meta-data to assets without having to create so many files?

    I understand the complexity of changing an existing system, just wanted to let you know my thoughts on workflow optimization as a long-time Unity user.

    1. Hi hannes, you should only backup the Assets, Packages and ProjectSettings folders, and possibly use GIT, instead of dropbox

    2. As per documentation, if you backup an Unity project, you should ignore the Library folder, which is the one that holds all those gazillions of imported files:
      https://docs.unity3d.com/Manual/ExternalVersionControlSystemSupport.html

    3. Hi Hannes,
      With regard to the large project size, Jonas Drewsen actually presented an upcoming feature which we call OnDemand mode, here’s a link with the time set to where he describes the problem and what we’re doing to address it: https://youtu.be/VF-Qe-0zXlc?t=1032

      With OnDemand you would actually get something even better than compression, which is no imported file at all! You would only import your full project, for example, when you build your game. If you don’t build your game locally, then you’ll never have the project imported completely, only what you’re working on, and the things which that depends on (i.e. if a model depends on a texture then the texture will be imported along with it).

  4. Will this affect the “Local Identifier In File” GUID? We use that property to track the persistant GUID of each GO (using the Transform’s GUID).

    If so, will Unity expose access to a more robust persistant GUID system for us to use?

    1. It will not affect the behaviour of local identifier in file.

      That said Unity 19.3 now has GlobalObjectIdentifier which is very useful for uniquely identifying a specific object with a stable persistent ID.

  5. hi – will this be coming to 2019.2 pls?
    we need to support iOS9 and 2019.3 is iOS10 only. thx

    1. Charles Beauchemin

      11月 7, 2019 4:33 pm 返信

      Once a version is released, only bugfixes are added to the version. We don’t backport new features.

      1. Ok thanks. Do you possibly know why iOS 10 is a 2019.3 requirement?

        1. JC Cimetiere

          11月 7, 2019 6:30 pm

          Hi Warren,
          iOS 9 market share worldwide has been shrinking to such a low level that it doesn’t make sense for us to continue supporting some optimizations & workarounds that were in Unity for this specific version. We can clean up the code and focus our attention to the newer versions.