Search Unity

Massively reduce Player build time and data size by allowing developers to control which Shader variants are handled by the Unity Shader compiler and included in the Player data.

Player build time and data size increase along with the complexity of your project because of the rising number of shader variants.

With scriptable shader variants stripping, introduced in 2018.2 beta, you can manage the number of shader variants generated and therefore drastically reduce Player build time and data size.

This feature allows you to strip all the shader variants with invalid code paths, strip shader variants for unused features or create shader build configurations such as “debug” and “release” without affecting iteration time or maintenance complexity.

In this blog post, we first define some of the terms we use. Then we focus on the definition of shader variants to explain why we can generate so many. This is followed by a description of automatic shader variants stripping and how scriptable shader variants stripping is implemented in the Unity shader pipeline architecture. Then, we look at the scriptable shader variants stripping API before discussing results on the Fountainbleau demo and concluding with some tips on writing stripping scripts.

Learning scriptable shader variants stripping is not a trivial undertaking, but it can lead to a massive increase in team efficiency!

Concepts

To understand the scriptable shader variants stripping feature it is important to have a precise understanding of the different concepts involved.

  • Shader asset: The full file source code with properties, sub-shader, passes, and HLSL.
  • Shader snippet: The HLSL input code with dependencies for a single shader stage.
  • Shader stage: A specific stage in the GPU rendering pipeline, typically a vertex shader stage and a fragment shader stage.
  • Shader keyword: A preprocessor identifier for compile-time branches across shaders.
  • Shader keyword set: A specific set of shader keywords identifying a particular code path.
  • Shader variant: The platform-specific shader code generated by the Unity shader compiler, for a single shader stage for a specific graphics tier, pass, shader keyword set, etc.
  • Uber shader: A shader source that can produce many shader variants.

In Unity, uber shaders are managed by ShaderLab sub shaders, passes, and shader types as well as the  #pragma multi_compile and #pragma shader_feature preprocessor directives.

Counting the number of shader variants generated

To use scriptable shader variant stripping, you need a clear understanding of what a shader variant is, and how shader variants are generated by the shader build pipeline. The number of shader variants generated is directly proportional to the build time and the Player shader variant data size. A shader variant is one output of the shader build pipeline.

Shader keywords are one of the elements that cause the generation of shader variants. An unconsidered use of shader keywords can quickly lead to a shader variants count explosion and therefore extremely long build time.

To see how shader variants are generated, the following simple shader couns how many shader variants it produces:

The total number of shader variants in a project is deterministic and given by the following equation:

The following trivial ShaderVariantStripping example brings clarity to this equation. It’s a single shader which simplifies the equation as following:Similarly, this shader has a single sub shader and a single pass which further simplifies the equation into:

 

 

Keywords in the equation refers to both platform and shader keywords. A graphics tier is a specific platform keyword set combination.

The ShaderVariantStripping/Pass has two multi compile directives. The first directive defines 4 keywords (COLOR_ORANGE, COLOR_VIOLET, COLOR_GREEN, COLOR_GRAY) and the second directive defines 3 keywords (OP_ADD, OP_MUL, OP_SUB). Finally, the pass defines 2 shader stages: a vertex shader stage and a fragment shader stage.

This shader variant total is given for a single supported graphics API. However, for each supported graphics API in the project, we need a dedicated set of shader variants. For example, if we build an Android Player that supports both OpenGL ES 3 and Vulkan, we need two sets of shader variants. As a result, the Player build time and shader data size are directly proportional to the number of supported graphics APIs.

Shader build pipeline

The shader compilation pipeline in Unity is a black box where each shader in the project is parsed to extract shader snippets before collecting the variant preprocessing instructions, such as multi_compile and shader_feature. This produces a list of compilation parameters, one per shader variant.

These compilation parameters include the shader snippet, the graphics tier, the shader type, the shader keyword set, the pass type and name. Each of the set compilation parameters are used to produce a single shader variant.

Consequently, Unity executes an automatic shader variant stripping pass based on two heuristics. Firstly, stripping is based on the Project Settings, for example, if Virtual Reality Supported is disabled then VR shader variants are systematically stripped. Second, the automatic stripping is based on the configuration of Shader Stripping section of the Graphics Settings.

Automatic shader variants stripping options in the GraphicsSettings.

Automatic shader variants stripping is based on build time restrictions. Unity can’t automatically select only the necessary shader variants at build time because those shader variants depend on runtime C# execution. For example, if a C# script adds a point light but there were no point lights at build time, then there is no way for the shader build pipeline to figure out that the Player would need a shader variant that does point light shading.

Here’s a list of shader variants with enabled keywords that get stripped automatically:

Lightmap modes: LIGHTMAP_ON, DIRLIGHTMAP_COMBINED, DYNAMICLIGHTMAP_ON, LIGHTMAP_SHADOW_MIXING, SHADOWS_SHADOWMASK

Fog modes: FOG_LINEAR, FOG_EXP, FOG_EXP2

Instancing Variants: INSTANCING_ON

Furthermore, when Virtual Reality support is disabled, the shader variants with the following built-in enabled keywords are stripped:

STEREO_INSTANCING_ON, STEREO_MULTIVIEW_ON, STEREO_CUBEMAP_RENDER_ON, UNITY_SINGLE_PASS_STEREO

When automatic stripping is done, the shader build pipeline uses the remaining compilation parameter sets to schedule shader variant compilation in parallel, launching as many simultaneous compilations as the platform has CPU core threads.

Here’s a visual representation of that process:

Shader pipeline architecture with scriptable shader variant stripping integration in orange.

In Unity 2018.2 beta, the shader pipeline architecture introduces a new stage right before the shader variant compilation scheduling, allowing users to control the shader variant compilation. This new stage is exposed via C# callbacks to user code, and each callback is executed per shader snippet.

Scriptable shader variant stripping API

As an example, the following script enables stripping of all the shader variants that would be associated with a “DEBUG” configuration, identified by a “DEBUG” keyword used in development Player build.

OnProcessShader is called right before the scheduling of the shader variant compilation.

Each combination of a Shader, a ShaderSnippetData and ShaderCompilerData instances is an identifier for a single shader variant that the shader compiler will produce. To strip that shader variant, we only need to remove it from the ShaderCompilerData list.

Every single shader variant that the shader compiler should generate will appear in this callback. When working on scripting the shader variants stripping, you need to first figure out which variants need removing, because they’re not useful for the project.

Results

Shader variants stripping for a render pipeline

One use case for the scriptable shader variants stripping is to systematically strip invalid shader variants of a render pipeline due to the various combinations of shader keywords.

A shader variants stripping script included in the HD render pipeline allows you to systematically reduce the build time and size of a project using the HD render pipeline. This script applies to the following shaders:

HDRenderPipeline/Lit
HDRenderPipeline/LitTessellation
HDRenderPipeline/LayeredLit
HDRenderPipeline/LayeredLitTessellation

The script produces the following results:

Unstripped Stripped
Player Data Shader Variant Count 24350 (100%) 12122 (49.8%)
Player Data Size on disk 511 MB 151 MB
Player Build Time 4864 seconds 1356 seconds

Screenshot of the Fontainebleau Photogrammetry demo using the HD Render Pipeline from the standard PlayStation 4 1920×1080 resolution.

Furthermore, the Lightweight render pipeline for Unity 2018.2 has a UI to automatically feed a stripping script that can automatically strip up to 98% of the shader variants which we expect to be particularly valuable for mobile projects.

Shader variants stripping for a project

Another use case is a script to strip all the rendering features of a render pipeline that are not used for a specific project. Using an internal test demo for the Lightweight rendering pipeline, we got the following results for the entire project:

Unstripped Stripped
Player Data Shader Variant Count 31080 7056
Player Data Size on disk 121 116
Player Build Time 839 seconds 286 seconds

As we can see, using scriptable shader variant stripping can lead to significant results and with more work on the stripping script we could go even further.

Screenshot of a Lightweight pipeline demo.

Tips on writing shader variants stripping code

Improving shader code design

A project may quickly run into a shader variants count explosion, leading to unsustainable compilation time and Player data size. Scriptable shader stripping helps deal with this issue, but you should reevaluate how you are using shader keywords to generate more relevant shader variants. We can rely on the #pragma skip_variants to test unused keywords in the editor.

For example, in ShaderStripping/Color Shader the preprocessing directives are declared with the following code:

This approach implies that all the combinations of color keywords and operator keywords will be generated.

Let’s say we want to render the following scene:

COLOR_ORANGE + OP_ADD, COLOR_VIOLET + OP_MUL, COLOR_GREEN + OP_MUL.

First, we should make sure that every keyword is actually useful. In this scene COLOR_GRAY and OP_SUB are never used. If we can guarantee these keywords are never used, then we should remove them.

Second, we should combine keywords that effectively produce a single code path. In this example, the ‘add’ operation is always used with the ‘orange’ color exclusively. So we can combine them in a single keyword and refactor the code as shown below.

Of course, it’s not always possible to refactor keywords. In these cases, scriptable shader variants stripping is a valuable tool!

Using callbackOrder to strip shader variants in multiple steps

For each snippet, all the shader variant stripping scripts are executed. We can order the scripts’ execution by ordering the value returned by the callbackOrder member function. The shader build pipeline will execute the callbacks in order of increasing callbackOrder, so lowest first and highest last.

A use case for using multiple shader stripping scripts is to separate the scripting per purpose. For example:

  • Script 1: Systematically strips all the shader variants with invalid code paths.
  • Script 2: Strips all the debug shader variants.
  • Script 3: Strips all the shader variants in the code base that are not necessary for the current project.
  • Script 4: Logs the remaining shader variants and strips them all for fast iteration time on the stripping scripts.

Process for writing a shader variants stripping script

Shader variants stripping is extremely powerful but requires a lot of work to achieve good results.

  1. In the Project view, filter for all shaders.
  2. Select a shader and, in the Inspector, click Show to open the list of keywords / variants of that shader. There will be a list of keywords that are always included in the build.
  3. Make sure that you know which specific graphics features the project uses.
  4. Check whether the keywords are used in all shader stages. Only one variant is necessary for stages that don’t use these keywords.
  5. Strip shader variants in the script.
  6. Verify the visuals in the build.
  7. Repeat steps 2 – 6 for each shader.

Download the sample project

The example project used to illustrate this blog post can be downloaded here. It requires Unity 2018.2.0b1.

Learn more about optimizing binary deployment size at Unite Berlin

Come to Jonas Echterhoff’s June 21 talk and learn about all the new tools that give you more control over what ends up in your build!

20 コメント

コメントの配信登録

コメント受付を終了しました。

  1. This looks like a very nice thing to have, but isn’t this what #pragma shader_feature is supposed to take care of for you? All the examples in the post uses multi_compile, but according to the manual the whole point of using shader_feature instead is that unused features are allegedly stripped at build time.

    It’d be great with an example in the context of #pragma shader_feature, or at least an explanation why we need to set up shader stripping manually instead of using shader_feature directives.

  2. François Dujardin

    5月 20, 2018 10:59 am

    Great post! Thanks!
    There’s a little typo about shader variant count example: 2x3x4=24, not 12.

  3. How about also having some more UI-enabled options too (like saying I am not going to use lights of type x/y/z if you can not know the answer) and display the shader counts before/after, so not everything requires editor scripts or at least you can do a little more [quickly] without scripting?

    1. See section “Shader variants stripping for a render pipeline”, this is being done for scriptable render pipelines.

  4. Vladislav Ivanchuk

    5月 14, 2018 9:42 pm

    Thanks, it really nice feature!!!

    What about iOS Metal shader compilation time? I mean metal shader compiler really slow, base shareds take 50-300 ms to compile on iPhone 6

    Do you think about asynk compile make possible?

    1. The gains using scriptable shader variants stripping is proportional to the number of shader variants compiled. If compiling a single shader variants is really slow, like on Metal, the gains will be particularly significant.

  5. How would you possibly ever determine
    A) all the possible keywords being generated for 31,000 shaders.
    B) which ones you actually do and don’t use?

    That sounds like an incredibly daunting task. Is there no way the compiler can make these determinations automatically?

    1. Yes this is a huge task and this is why the blog post is so detailed. The big issue is that the actual used shader variants depends on the actual C# execution.

      It’s not necessary to strip everything. The C# API allows logging all the shader variants at build time. This is a good start to remove shader variants that seems obviously unused.

  6. Don’t you think pseudocode would have been a better choice than sigma notation – considering your audience?

  7. I’m interested in a Shader (Lit) for Terran and a new system for creating Terran… one question: When?
    UPD: Sorry for my English.

  8. Robert Cummings

    5月 14, 2018 4:10 pm

    Nice! very helpful. I understand this isn’t in 1.1.5 HD preview, but 1.1.6?

  9. Helpful post! (The link of the example project is dead btw)

    1. Yes, this is platform independent

    2. Euhm… something went wrong, I’ll fix it on Tuesday.

  10. Project link at the end of blog post doesn’t work.

    1. Euhm… something went wrong, I’ll fix it on Tuesday.

  11. Could you take advantage of this on WebGL?

    1. Yes, this is platform independent