Search Unity

Stripping scriptable shader variants

May 14, 2018 in Engine & platform | 12 min. read
Share

Is this article helpful for you?

Thank you for your feedback!

Massively reduce Player build time and data size by allowing developers to control which Shader variants are handled by the Unity Shader compiler and included in the Player data.

Player build time and data size increase along with the complexity of your project because of the rising number of shader variants.

With scriptable shader variants stripping, introduced in 2018.2 beta, you can manage the number of shader variants generated and therefore drastically reduce Player build time and data size.

This feature allows you to strip all the shader variants with invalid code paths, strip shader variants for unused features or create shader build configurations such as “debug” and “release” without affecting iteration time or maintenance complexity.

In this blog post, we first define some of the terms we use. Then we focus on the definition of shader variants to explain why we can generate so many. This is followed by a description of automatic shader variants stripping and how scriptable shader variants stripping is implemented in the Unity shader pipeline architecture. Then, we look at the scriptable shader variants stripping API before discussing results on the Fountainbleau demo and concluding with some tips on writing stripping scripts.

Learning scriptable shader variants stripping is not a trivial undertaking, but it can lead to a massive increase in team efficiency!

Concepts

To understand the scriptable shader variants stripping feature it is important to have a precise understanding of the different concepts involved.

  • Shader asset: The full file source code with properties, sub-shader, passes, and HLSL.
  • Shader snippet: The HLSL input code with dependencies for a single shader stage.
  • Shader stage: A specific stage in the GPU rendering pipeline, typically a vertex shader stage and a fragment shader stage.
  • Shader keyword: A preprocessor identifier for compile-time branches across shaders.
  • Shader keyword set: A specific set of shader keywords identifying a particular code path.
  • Shader variant: The platform-specific shader code generated by the Unity shader compiler, for a single shader stage for a specific graphics tier, pass, shader keyword set, etc.
  • Uber shader: A shader source that can produce many shader variants.

In Unity, uber shaders are managed by ShaderLab sub shaders, passes, and shader types as well as the  #pragma multi_compile and #pragma shader_feature preprocessor directives.

Counting the number of shader variants generated

To use scriptable shader variant stripping, you need a clear understanding of what a shader variant is, and how shader variants are generated by the shader build pipeline. The number of shader variants generated is directly proportional to the build time and the Player shader variant data size. A shader variant is one output of the shader build pipeline.

Shader keywords are one of the elements that cause the generation of shader variants. An unconsidered use of shader keywords can quickly lead to a shader variants count explosion and therefore extremely long build time.

To see how shader variants are generated, the following simple shader couns how many shader variants it produces:

Shader "ShaderVariantsStripping"
{
	SubShader
	{
		Pass
		{
			Name "ShaderVariantsStripping/Pass"

			CGPROGRAM
			#pragma vertex vert
			#pragma fragment frag
			#pragma multi_compile COLOR_ORANGE COLOR_VIOLET COLOR_GREEN COLOR_GRAY
			#pragma multi_compile OP_ADD OP_MUL OP_SUB

			struct appdata
			{
				float4 vertex : POSITION;
				float2 uv : TEXCOORD0;
			};

			struct v2f
			{
				float2 uv : TEXCOORD0;
				float4 vertex : SV_POSITION;
			};

			sampler2D _MainTex;
			float4 _MainTex_ST;

			v2f vert (appdata v)
			{
				v2f o;
				o.vertex = UnityObjectToClipPos(v.vertex);
				o.uv = v.uv;
				return o;
			}

			fixed4 get_color()
			{
				#if defined(COLOR_ORANGE)
					return fixed4(1.0, 0.5, 0.0, 1.0);
				#elif defined(COLOR_VIOLET)
					return fixed4(0.8, 0.2, 0.8, 1.0);
				#elif defined(COLOR_GREEN)
					return fixed4(0.5, 0.9, 0.3, 1.0);
				#elif defined(COLOR_GRAY)
					return fixed4(0.5, 0.9, 0.3, 1.0);
				#else
					#error "Unknown 'color' keyword"
				#endif
			}

			fixed4 frag (v2f i) : SV_Target
			{
				fixed4 diffuse = tex2D(_MainTex, i.uv);

				fixed4 color = get_color();

				#if defined(OP_ADD)
					return diffuse + color;
				#elif defined(OP_MUL)
					return diffuse * color;
				#elif defined(OP_SUB)
					return diffuse - color;
				#else
					#error "Unknown 'op' keyword"
				#endif
			}
			ENDCG
		}
	}
}

The total number of shader variants in a project is deterministic and given by the following equation:

The following trivial ShaderVariantStripping example brings clarity to this equation. It’s a single shader which simplifies the equation as following:

Similarly, this shader has a single sub shader and a single pass which further simplifies the equation into:

Keywords in the equation refers to both platform and shader keywords. A graphics tier is a specific platform keyword set combination.

The ShaderVariantStripping/Pass has two multi compile directives. The first directive defines 4 keywords (COLOR_ORANGE, COLOR_VIOLET, COLOR_GREEN, COLOR_GRAY) and the second directive defines 3 keywords (OP_ADD, OP_MUL, OP_SUB). Finally, the pass defines 2 shader stages: a vertex shader stage and a fragment shader stage.

This shader variant total is given for a single supported graphics API. However, for each supported graphics API in the project, we need a dedicated set of shader variants. For example, if we build an Android Player that supports both OpenGL ES 3 and Vulkan, we need two sets of shader variants. As a result, the Player build time and shader data size are directly proportional to the number of supported graphics APIs.

Shader build pipeline

The shader compilation pipeline in Unity is a black box where each shader in the project is parsed to extract shader snippets before collecting the variant preprocessing instructions, such as multi_compile and shader_feature. This produces a list of compilation parameters, one per shader variant.

These compilation parameters include the shader snippet, the graphics tier, the shader type, the shader keyword set, the pass type and name. Each of the set compilation parameters are used to produce a single shader variant.

Consequently, Unity executes an automatic shader variant stripping pass based on two heuristics. Firstly, stripping is based on the Project Settings, for example, if Virtual Reality Supported is disabled then VR shader variants are systematically stripped. Second, the automatic stripping is based on the configuration of Shader Stripping section of the Graphics Settings.

Automatic shader variants stripping options in the GraphicsSettings.

Automatic shader variants stripping is based on build time restrictions. Unity can’t automatically select only the necessary shader variants at build time because those shader variants depend on runtime C# execution. For example, if a C# script adds a point light but there were no point lights at build time, then there is no way for the shader build pipeline to figure out that the Player would need a shader variant that does point light shading.

Here’s a list of shader variants with enabled keywords that get stripped automatically:

Lightmap modes: LIGHTMAP_ON, DIRLIGHTMAP_COMBINED, DYNAMICLIGHTMAP_ON, LIGHTMAP_SHADOW_MIXING, SHADOWS_SHADOWMASK

Fog modes: FOG_LINEAR, FOG_EXP, FOG_EXP2

Instancing Variants: INSTANCING_ON

Furthermore, when Virtual Reality support is disabled, the shader variants with the following built-in enabled keywords are stripped:

STEREO_INSTANCING_ON, STEREO_MULTIVIEW_ON, STEREO_CUBEMAP_RENDER_ON, UNITY_SINGLE_PASS_STEREO

When automatic stripping is done, the shader build pipeline uses the remaining compilation parameter sets to schedule shader variant compilation in parallel, launching as many simultaneous compilations as the platform has CPU core threads.

Here’s a visual representation of that process:

Shader pipeline architecture with scriptable shader variant stripping integration in orange.

In Unity 2018.2 beta, the shader pipeline architecture introduces a new stage right before the shader variant compilation scheduling, allowing users to control the shader variant compilation. This new stage is exposed via C# callbacks to user code, and each callback is executed per shader snippet.

Scriptable shader variant stripping API

As an example, the following script enables stripping of all the shader variants that would be associated with a “DEBUG” configuration, identified by a “DEBUG” keyword used in development Player build.

using System.Collections.Generic;
using UnityEditor;
using UnityEditor.Build;
using UnityEditor.Rendering;
using UnityEngine;
using UnityEngine.Rendering;

// Simple example of stripping of a debug build configuration
class ShaderDebugBuildProcessor : IPreprocessShaders
{
    ShaderKeyword m_KeywordDebug;

    public ShaderDebugBuildProcessor()
    {
        m_KeywordDebug = new ShaderKeyword("DEBUG");
    }

    // Multiple callback may be implemented.
    // The first one executed is the one where callbackOrder is returning the smallest number.
    public int callbackOrder { get { return 0; } }

    public void OnProcessShader(
        Shader shader, ShaderSnippetData snippet, IList<ShaderCompilerData> shaderCompilerData)
    {
        // In development, don't strip debug variants
        if (EditorUserBuildSettings.development)
            return;

        for (int i = 0; i < shaderCompilerData.Count; ++i)
        {
            if (shaderCompilerData[i].shaderKeywordSet.IsEnabled(m_KeywordDebug))
            {
                shaderCompilerData.RemoveAt(i);
                --i;
            }
        }
    }
}

OnProcessShader is called right before the scheduling of the shader variant compilation.

Each combination of a Shader, a ShaderSnippetData and ShaderCompilerData instances is an identifier for a single shader variant that the shader compiler will produce. To strip that shader variant, we only need to remove it from the ShaderCompilerData list.

Every single shader variant that the shader compiler should generate will appear in this callback. When working on scripting the shader variants stripping, you need to first figure out which variants need removing, because they’re not useful for the project.

Results

Shader variants stripping for a render pipeline

One use case for the scriptable shader variants stripping is to systematically strip invalid shader variants of a render pipeline due to the various combinations of shader keywords.

A shader variants stripping script included in the HD render pipeline allows you to systematically reduce the build time and size of a project using the HD render pipeline. This script applies to the following shaders:

HDRenderPipeline/Lit
HDRenderPipeline/LitTessellation
HDRenderPipeline/LayeredLit
HDRenderPipeline/LayeredLitTessellation

The script produces the following results:

UnstrippedStripped
Player Data Shader Variant Count24350 (100%)12122 (49.8%)
Player Data Size on disk511 MB151 MB
Player Build Time4864 seconds1356 seconds
Screenshot of the Fontainebleau Photogrammetry demo using the HD Render Pipeline from the standard PlayStation 4 1920x1080 resolution.

Furthermore, the Lightweight render pipeline for Unity 2018.2 has a UI to automatically feed a stripping script that can automatically strip up to 98% of the shader variants which we expect to be particularly valuable for mobile projects.

Shader variants stripping for a project

Another use case is a script to strip all the rendering features of a render pipeline that are not used for a specific project. Using an internal test demo for the Lightweight rendering pipeline, we got the following results for the entire project:

UnstrippedStripped
Player Data Shader Variant Count310807056
Player Data Size on disk121116
Player Build Time839 seconds286 seconds

As we can see, using scriptable shader variant stripping can lead to significant results and with more work on the stripping script we could go even further.

Screenshot of a Lightweight pipeline demo.

Tips on writing shader variants stripping code

Improving shader code design

A project may quickly run into a shader variants count explosion, leading to unsustainable compilation time and Player data size. Scriptable shader stripping helps deal with this issue, but you should reevaluate how you are using shader keywords to generate more relevant shader variants. We can rely on the #pragma skip_variants to test unused keywords in the editor.

For example, in ShaderStripping/Color Shader the preprocessing directives are declared with the following code:

#pragma multi_compile COLOR_ORANGE COLOR_VIOLET COLOR_GREEN COLOR_GRAY // color keywords
#pragma multi_compile OP_ADD OP_MUL OP_SUB // operator keywords

This approach implies that all the combinations of color keywords and operator keywords will be generated.

Let’s say we want to render the following scene:

COLOR_ORANGE + OP_ADD, COLOR_VIOLET + OP_MUL, COLOR_GREEN + OP_MUL.

First, we should make sure that every keyword is actually useful. In this scene COLOR_GRAY and OP_SUB are never used. If we can guarantee these keywords are never used, then we should remove them.

Second, we should combine keywords that effectively produce a single code path. In this example, the ‘add’ operation is always used with the ‘orange’ color exclusively. So we can combine them in a single keyword and refactor the code as shown below.

#pragma multi_compile ADD_COLOR_ORANGE MUL_COLOR_VIOLET MUL_COLOR_GREEN

#if defined(ADD_COLOR_ORANGE)
	#define COLOR_ORANGE
	#define OP_ADD
#elif defined(MUL_COLOR_VIOLET)
	#define COLOR_VIOLET
	#define OP_MUL
#elif defined(MUL_COLOR_GREEN)
	#define COLOR_GREEN
	#define OP_MUL
#endif

Of course, it’s not always possible to refactor keywords. In these cases, scriptable shader variants stripping is a valuable tool!

Using callbackOrder to strip shader variants in multiple steps

For each snippet, all the shader variant stripping scripts are executed. We can order the scripts’ execution by ordering the value returned by the callbackOrder member function. The shader build pipeline will execute the callbacks in order of increasing callbackOrder, so lowest first and highest last.

A use case for using multiple shader stripping scripts is to separate the scripting per purpose. For example:

  • Script 1: Systematically strips all the shader variants with invalid code paths.
  • Script 2: Strips all the debug shader variants.
  • Script 3: Strips all the shader variants in the code base that are not necessary for the current project.
  • Script 4: Logs the remaining shader variants and strips them all for fast iteration time on the stripping scripts.

Process for writing a shader variants stripping script

Shader variants stripping is extremely powerful but requires a lot of work to achieve good results.

  1. In the Project view, filter for all shaders.
  2. Select a shader and, in the Inspector, click Show to open the list of keywords / variants of that shader. There will be a list of keywords that are always included in the build.
  3. Make sure that you know which specific graphics features the project uses.
  4. Check whether the keywords are used in all shader stages. Only one variant is necessary for stages that don’t use these keywords.
  5. Strip shader variants in the script.
  6. Verify the visuals in the build.
  7. Repeat steps 2 - 6 for each shader.

Download the sample project

The example project used to illustrate this blog post can be downloaded here. It requires Unity 2018.2.0b1.

Learn more about optimizing binary deployment size at Unite Berlin

Come to Jonas Echterhoff's June 21 talk and learn about all the new tools that give you more control over what ends up in your build!

May 14, 2018 in Engine & platform | 12 min. read

Is this article helpful for you?

Thank you for your feedback!

Related Posts