Ahem.

So yeah, a bunch of us were at SIGGRAPH 2011 in lovely Vancouver, doing talks, meeting people and going to super interesting sessions. Me & ReJ did a long technical session about optimizing graphics for mobile platforms. GPU architectures, shader optimization details, case studies from actual Unity games and so on.

Here it is: Fast Mobile Shaders (17MB pdf).

Enjoy!

But is it secure?

“Bububut, waitaminnit! Native code is not secure by definition” you say. Turns out, that isn’t necessarily true. With a specially massaged compiler, some runtime support and careful native code validation it is possible to ensure native code, when ran in the browser, can’t cause harm to user’s machine. I suggest taking a look at the original NaCl for x86 paper and more recently, how similar techniques would apply to ARM CPUs.

But what can you do with it?

So that’s great. It means it is possible to take C/C++ code, compile it with NaCl SDK (a gcc derived toolchain) and have it run in the browser. We can make a loop in C that multiplies a ton of floating point numbers, and it will run at native speed. That’s wonderful, except you can’t really do much interesting stuff with your own C code in isolation…

You need access to the hardware and/or OS. As game developers, we need pixels to appear on the screen. Preferably lots of them, with the help of something like a GPU. Audio waves to come out of the speakers. Mouse moves and keyboard presses to translate to some fancy actions. Post a high score to the internets. And so on.

NaCl surely can’t just allow my C code to call Direct3DCreate9 and run with it, while keeping the promise of “it’s secure”? Or a more extreme case, FILE* f = fopen("/etc/passwd", "rt");?!

And that’s true; NaCl does not allow you to use completely arbitrary APIs. It has it’s own set of APIs to interface with “the system”.

Ok, how do I interface with the system?

…and that’s where the current state of NaCl gets a bit confusing.

Initially Google developed an improved “browser plugin model” and called it Pepper. This Pepper thing would then take care of actually putting your code into the browser. Starting it up, tearing it down, controlling repaints, processing events and so on. But then apparently they realized that building on top of a decade-old Netscape plugin API (NPAPI) isn’t going to really work, so they developed Pepper2 or PPAPI (Pepper Plugin API) which ditches NPAPI completely. To write a native client plugin, you only interface with PPAPI.

So some of the pages on the internets reference the “old API” (which is gone as far as I can see), and some others reference the new one. It does not help that Native Client’s own documentation are scattered around in Chromium, NaCl, NaCl SDK and PPAPI sites. Seriously, it’s a mess, with seemingly no high level, up to date “introduction” page that tells what exactly PPAPI can and can’t do. Edit: I’m told that the definitive entry point to NaCl right now is this page: http://code.google.com/chrome/nativeclient/ which clears up some mess.

Here’s what I think it can do

Note: we have an in-progress Unity NaCl port using this PPAPI. However, I am not working on it, so my knowledge may or may not be true. Take everything with a grain of NaCl

Most of things below found by poking around at PPAPI source tree, and by looking into Unity’s NaCl platform dependent bits.

Graphics

PPAPI provides an OpenGL ES 2.0 implementation for your 3D needs. You need to setup the context and initial surfaces via PPAPI (ppapi/cpp/dev/context_3d_dev.h, ppapi/cpp/dev/surface_3d_dev.h) – similar to what you’d use EGL on other platforms for – and beyond that you just include GLES2/gl2.h, GLES2/gl2ext.h and call ye olde GLES2.0 functions.

Behind the scenes, all your GLES2.0 calls will be put into a command buffer and transferred to actual “3D server” process for consuming them. Chrome splits up itself into various processes like that for security reasons — so that each process has the minimum set of privileges, and a crash or a security exploit in one of them can’t easily transfer over to other parts of the browser.

Audio

For audio needs, PPAPI provides a simple buffer based API in ppapi/cpp/audio_config.h and ppapi/cpp/audio.h. Your own callback will be called whenever audio buffer needs to be filled with new samples. That means you do all sound mixing yourself and just fill in the final buffer.

Input

Your plugin instance (subclass of pp::Instance) will get input events via HandleInputEvent virtual function override. Each event is a simple PPInputEvent struct and can represent keyboard & mouse. No support for gamepads or touch input so far, it seems.

Other stuff

Doing WWW requests is possible via ppapi/cpp/url_loader.h and friends.

Timer & time queries via ppapi/cpp/core.h (e.g. pp::Module::Get()->core()->CallOnMainThread(...)).

And, well, a bunch of other stuff is there, like ability to rasterize blocks of text into bitmaps, pop up file selection dialogs, use the browser to decode video streams and so on. Everything – or almost everything – is there to make it possible to do games on it.

Summary

Like Chad says, it would be good to end “thou shalt only use Javascript” on the web. Javascript is a very nice language – especially considering how it came into existence – but forcing it on everyone is quite silly. And no matter how hard V8/JägerMonkey/Nitro folks are trying, it is very, very hard to beat performance of a simple, static, compiled language (like C) that has direct access to memory and the programmer is in almost full control of both the code flow and the memory layout. Steve rightly points out that even if for some tasks a super-optimized Javascript engine will approach the speed of C, it will burn much more energy to do so — a very important aspect in the increasingly mobile world.

Native Client does give some hope that there will be a way to run native code, at native speeds, in the browser, without compromising on security. Let it happen.

Precision of Shader Computations
Unity 3.2 makes much better use of different precision shader computations in OpenGL ES 2.0 shaders. This alone results in a 50%-150% speed increase on more complex shaders. Our GLSL Translator & Optimizer has been extended so that float/half/fixed types in Cg map to highp/mediump/lowp precision in GLSL, and we’ve updated all built-in shaders to use appropriate precision.

Normal Mapping
We’ve made several tweaks to make normal mapping faster. On desktops, we use “DXT5nm” compression for normal maps, where two normal components are stored and third is computed in the shader – this works great because it allows using DXT5 on the normalmaps. However, the extra computation is not very good for mobile platforms, so we’ve changed the normal map compression scheme there. We’ve also implement assembly optimized skinning for normal mapped meshes — this is now 4x faster than before.

Built-In Shaders
Unity 3.2 will ship with new shaders that are optimized variants of existing built-in shaders. You’ll be able to find them under the “Mobile” category (but they work on other platforms as well!). These new shaders have a few limitations, but the upside is increased performance — the Mobile/Bumped Specular, for example, is 5.2x faster than the Bumped Specular currently shipping in Unity 3.1 (tested on iOS).

Other Optimizations
There are a number of other improvements that make Unity-created mobile content run faster in 3.2. For example, we’ve optimized the internals of the OpenGL ES 2.0 renderer for lower CPU overhead. We’ve also changed how Unity handles alpha-testing so that those objects are rendered after all fully opaque ones. These and other tweaks and adjustments are all designed to help you get the most out of your mobile game.

Warning: technical post ahead!

Shading Languages

There are two widely used shader languages out there: HLSL (or Cg, which is the same for practical purposes) and GLSL. Cg/HLSL is used by Direct3D, Xbox 360 and PS3. GLSL is used by mobile platforms (OpenGL ES 2.0), Mac OS X (OpenGL) and upcoming WebGL.

Traditionally in Unity shaders are written in Cg/HLSL (there’s an option to write GLSL if you want to), so we had to find a way to take Cg shader and produce GLSL shader out of it. So you could write a shader like:

// ...vertex shader omitted
sampler2D _MainTex;
half4 _Color;
half4 frag (float2 uv:TEXCOORD0) : COLOR {
    return tex2D (_MainTex, uv) * _Color;
}

…and somehow it would also work on mobile platforms on OpenGL ES 2.0. In short, you can do that in Unity 3. But if you want to know how that works, read on.

HLSL to GLSL Translator

The two shading languages are similar in principle, but there are lots of subtle differences that make a regexp-based converter not quite work (regular expression based parsers are rarely a good idea by the way).

So we took HLSL2GLSL, an existing open source project that ATI has made 4 years ago and seemingly abandoned. Turns out it needed some massaging to get it work, as well as a ton of bug fixes, missing features and general code cleanup. But hey, it seems to work now!

Our fork of this translator, named “hlsl2glslfork”, is here: http://code.google.com/p/hlsl2glslfork/
(you can probably tell that I’m not too good at picking fancy names)

So now we can translate Cg/HLSL shaders into GLSL, and we’re done? Turns out, not quite so.

GLSL Optimizer

The shaders produced by the translator work on mobile platforms (iOS/Android). However, some of them were running very slow. The OpenGL ES 2.0 drivers in the mobile platforms aren’t very good at optimizing shaders! Or rather, some of them are seriously bad at optimizing.

Suffice to say, something extremely simple like a particle shader was running 6 times slower than it should have. Six. Times. Ouch!

Almost by accident, I discovered that Mesa 3D guys are working on new GLSL compiler. I looked at the code and I liked it a lot; very hackable and “no bullshit” approach. So I took that Mesa’s GLSL compiler and made it output GLSL back after it has done all the optimizations (dead code removal, algebraic simplifications, constant propagation, constant folding, inlining and a bunch of other cryptic things that don’t mean anything to a normal person).

Here it is, named “GLSL Optimizer”: http://github.com/aras-p/glsl-optimizer
(here’s my fancy-name-choosing in action again)

The good news is that it solves the performance problems on iOS/Android platforms. Yay!

Further work

For mobile platforms, using appropriate precision qualifiers is very important for performance. If some value has low range and does not need high precision, using “lowp” (low precision) qualifier can potentially make the shader work much faster. In Unity 3.0 we do some automagic to put the appropriate precision, but we could be better at it. I’m experimenting with proper precision support in HLSL to GLSL translator and the GLSL Optimizer, so that fixed/half/float types in Cg/HLSL would come out as lowp/mediump/highp in the resulting GLSL… we’ll see how that turns out.

Maybe at some point in the future it would make sense to have the opposite translation tool? So one could write GLSL shaders, and they would be converted into Cg/HLSL for the other platforms…

Oh, and next week Unity’s mobile tech. lead Renaldas “ReJ” Zioma is going to blog something technology related as well. Hear me ReJ? Now you have to do it!

Warning: a technical post ahead with almost no pictures!

Over a year ago I had a thought that “Shaders must die” (part 1, part 2, part 3). And what do you know – turns out we’re doing this in Unity 3. We call this Surface Shaders cause I’ve a suspicion “shaders must die” as a feature name wouldn’t have flied very far.

Idea

The main idea is that 90% of the time I just want to declare surface properties. This is what I want to say:

Hey, albedo comes from this texture mixed with this texture, and normal comes from this normal map. Use Blinn-Phong lighting model please, and don’t bother me again!

With the above, I don’t have to care whether this will be used in a forward or deferred rendering, or how various light types will be handled, or how many lights per pass will be done in a forward renderer, or how some indirect illumination SH probes will come in, etc. I’m not interested in all that! These dirty bits are job of rendering programmers, just make it work dammit!

This is not a new idea. Most graphical shader editors that make sense do not have “pixel color” as the final output node; instead they have some node that basically describes surface parameters (diffuse, specularity, normal, …), and all the lighting code is usually not expressed in the shader graph itself. OpenShadingLanguage is a similar idea as well (but because it’s targeted at offline rendering for movies, it’s much richer & more complex).

Example

Here’s a simple – but full & complete – Unity 3.0 shader that does diffuse lighting with a texture & a normal map.

  Shader "Example/Diffuse Bump" {
    Properties {
      _MainTex ("Texture", 2D) = "white" {}
      _BumpMap ("Bumpmap", 2D) = "bump" {}
    }
    SubShader {
      Tags { "RenderType" = "Opaque" }
      CGPROGRAM
      #pragma surface surf Lambert
      struct Input {
        float2 uv_MainTex;
        float2 uv_BumpMap;
      };
      sampler2D _MainTex;
      sampler2D _BumpMap;
      void surf (Input IN, inout SurfaceOutput o) {
        o.Albedo = tex2D (_MainTex, IN.uv_MainTex).rgb;
        o.Normal = UnpackNormal (tex2D (_BumpMap, IN.uv_BumpMap));
      }
      ENDCG
    }
    Fallback "Diffuse"
  }

Given pretty model & textures, it can produce pretty pictures! How cool is that?

I grayed out bits that are not really interesting (declaration of serialized shader properties & their UI names, shader fallback for older machines etc.). What’s left is Cg/HLSL code, which is then augmented by tons of auto-generated code that deals with lighting & whatnot.

This surface shader dissected into pieces:

• #pragma surface surf Lambert: this is a surface shader with main function “surf”, and a Lambert lighting model. Lambert is one of predefined lighting models, but you can write your own.
• struct Input: input data for the surface shader. This can have various predefined inputs that will be computed per-vertex & passed into your surface function per-pixel. In this case, it’s two texture coordinates.
• surf function: actual surface shader code. It takes Input, and writes into SurfaceOutput (a predefined structure). It is possible to write into custom structures, provided you use lighting models that operate on those structures. The actual code just writes Albedo and Normal to the output.

What is generated

Unity’s “surface shader code generator” would take this, generate actual vertex & pixel shaders, and compile them to various target platforms. With default settings in Unity 3.0, it would make this shader support:

• Forward renderer and Deferred Lighting (Light Pre-Pass) renderer.
• Objects with precomputed lightmaps and without.
• Directional, Point and Spot lights; with projected light cookies or without; with shadowmaps or without. Well ok, this is only for forward renderer because in Deferred Lighting the lighting happens elsewhere.
• For Forward renderer, it would compile in support for lights computed per-vertex and spherical harmonics lights computed per-object. It would also generate extra additive blended pass if needed for the case when additional per-pixel lights have to be rendered in separate passes.
• For Deferred Lighting, it would generate base pass that outputs normals & specular power; and a final pass that combines albedo with lighting, adds in any lightmaps or emissive lighting etc.
• It can optionally generate a shadow caster rendering pass (needed if custom vertex position modifiers are used for vertex shader based animation; or some complex alpha-test effects are done).

For example, here’s code that would be compiled for a forward-rendered base pass with one directional light, 4 per-vertex point lights, 3rd order SH lights; optional lightmaps (I suggest just scrolling down):

#pragma vertex vert_surf
#pragma fragment frag_surf
#pragma fragmentoption ARB_fog_exp2
#pragma fragmentoption ARB_precision_hint_fastest
#pragma multi_compile_fwdbase
#include "HLSLSupport.cginc"
#include "UnityCG.cginc"
#include "Lighting.cginc"
#include "AutoLight.cginc"
struct Input {
	float2 uv_MainTex : TEXCOORD0;
};
sampler2D _MainTex;
sampler2D _BumpMap;
void surf (Input IN, inout SurfaceOutput o)
{
	o.Albedo = tex2D (_MainTex, IN.uv_MainTex).rgb;
	o.Normal = UnpackNormal (tex2D (_BumpMap, IN.uv_MainTex));
}
struct v2f_surf {
  V2F_POS_FOG;
  float2 hip_pack0 : TEXCOORD0;
  #ifndef LIGHTMAP_OFF
  float2 hip_lmap : TEXCOORD1;
  #else
  float3 lightDir : TEXCOORD1;
  float3 vlight : TEXCOORD2;
  #endif
  LIGHTING_COORDS(3,4)
};
#ifndef LIGHTMAP_OFF
float4 unity_LightmapST;
#endif
float4 _MainTex_ST;
v2f_surf vert_surf (appdata_full v) {
  v2f_surf o;
  PositionFog( v.vertex, o.pos, o.fog );
  o.hip_pack0.xy = TRANSFORM_TEX(v.texcoord, _MainTex);
  #ifndef LIGHTMAP_OFF
  o.hip_lmap.xy = v.texcoord1.xy * unity_LightmapST.xy + unity_LightmapST.zw;
  #endif
  float3 worldN = mul((float3x3)_Object2World, SCALED_NORMAL);
  TANGENT_SPACE_ROTATION;
  #ifdef LIGHTMAP_OFF
  o.lightDir = mul (rotation, ObjSpaceLightDir(v.vertex));
  #endif
  #ifdef LIGHTMAP_OFF
  float3 shlight = ShadeSH9 (float4(worldN,1.0));
  o.vlight = shlight;
  #ifdef VERTEXLIGHT_ON
  float3 worldPos = mul(_Object2World, v.vertex).xyz;
  o.vlight += Shade4PointLights (
    unity_4LightPosX0, unity_4LightPosY0, unity_4LightPosZ0,
    unity_LightColor0, unity_LightColor1, unity_LightColor2, unity_LightColor3,
    unity_4LightAtten0, worldPos, worldN );
  #endif // VERTEXLIGHT_ON
  #endif // LIGHTMAP_OFF
  TRANSFER_VERTEX_TO_FRAGMENT(o);
  return o;
}
#ifndef LIGHTMAP_OFF
sampler2D unity_Lightmap;
#endif
half4 frag_surf (v2f_surf IN) : COLOR {
  Input surfIN;
  surfIN.uv_MainTex = IN.hip_pack0.xy;
  SurfaceOutput o;
  o.Albedo = 0.0;
  o.Emission = 0.0;
  o.Specular = 0.0;
  o.Alpha = 0.0;
  o.Gloss = 0.0;
  surf (surfIN, o);
  half atten = LIGHT_ATTENUATION(IN);
  half4 c;
  #ifdef LIGHTMAP_OFF
  c = LightingLambert (o, IN.lightDir, atten);
  c.rgb += o.Albedo * IN.vlight;
  #else // LIGHTMAP_OFF
  half3 lmFull = DecodeLightmap (tex2D(unity_Lightmap, IN.hip_lmap.xy));
  #ifdef SHADOWS_SCREEN
  c.rgb = o.Albedo * min(lmFull, atten*2);
  #else
  c.rgb = o.Albedo * lmFull;
  #endif
  c.a = o.Alpha;
  #endif // LIGHTMAP_OFF
  return c;
}

Of those 90 lines of code, 10 are your original surface shader code; the remaining 80 would have to be pretty much written by hand in Unity 2.x days (well ok, less code would have to be written because 2.x had less rendering features). But wait, that was only base pass of the forward renderer! It also generates code for additive pass, for deferred base pass, deferred final pass, optionally for shadow caster pass and so on.

So this should be an easier to write lit shaders (it is for me at least). I hope this will also increase the number of Unity users who can write shaders at least 3 times (i.e. to 30 up from 10!). It should be more future proof to accomodate changes to the lighting pipeline we’ll do in Unity next.

Predefined Input values

The Input structure can contain texture coordinates and some predefined values, for example view direction, world space position, world space reflection vector and so on. Code to compute them is only generated if they are actually used. For example, if you use world space reflection to do some cubemap reflections (as emissive term) in your surface shader, then in Deferred Lighting base pass the reflection vector will not be computed (since it does not output emission, so by extension does not need reflection vector).

As a small example, the shader above extended to do simple rim lighting:

  #pragma surface surf Lambert
  struct Input {
      float2 uv_MainTex;
      float2 uv_BumpMap;
      float3 viewDir;
  };
  sampler2D _MainTex;
  sampler2D _BumpMap;
  float4 _RimColor;
  float _RimPower;
  void surf (Input IN, inout SurfaceOutput o) {
      o.Albedo = tex2D (_MainTex, IN.uv_MainTex).rgb;
      o.Normal = UnpackNormal (tex2D (_BumpMap, IN.uv_BumpMap));
      half rim =
          1.0 - saturate(dot (normalize(IN.viewDir), o.Normal));
      o.Emission = _RimColor.rgb * pow (rim, _RimPower);
  }

Vertex shader modifiers

It is possible to specify custom “vertex modifier” function that will be called at start of the generated vertex shader, to modify (or generate) per-vertex data. You know, vertex shader based tree wind animation, grass billboard extrusion and so on. It can also fill in any non-predefined values in the Input structure.

My favorite vertex modifier? Moving vertices along their normals.

Custom Lighting Models

There are a couple simple lighting models built-in, but it’s possible to specify your own. A lighting model is nothing more than a function that will be called with the filled SurfaceOutput structure and per-light parameters (direction, attenuation and so on). Different functions would have to be called in forward & deferred rendering cases; and naturally the deferred one has much less flexibility. So for any fancy effects, it is possible to say “do not compile this shader for deferred”, in which case it will be rendered via forward rendering.

Example of wrapped-Lambert lighting model:

  #pragma surface surf WrapLambert
  half4 LightingWrapLambert (SurfaceOutput s, half3 dir, half atten) {
      dir = normalize(dir);
      half NdotL = dot (s.Normal, dir);
      half diff = NdotL * 0.5 + 0.5;
      half4 c;
      c.rgb = s.Albedo * _LightColor0.rgb * (diff * atten * 2);
      c.a = s.Alpha;
      return c;
  }
  struct Input {
      float2 uv_MainTex;
  };
  sampler2D _MainTex;
  void surf (Input IN, inout SurfaceOutput o) {
      o.Albedo = tex2D (_MainTex, IN.uv_MainTex).rgb;
  }

Behind the scenes

We’re using HLSL parser from Ryan Gordon’s mojoshader to parse the original surface shader code and infer some things from the abstract syntax tree mojoshader produces. This way we can figure out what members are in what structures, go over function prototypes and so on. At this stage some error checking is done to tell the user his surface function is of wrong prototype, or his structures are missing required members – which is much better than failing with dozens of compile errors in the generated code later.

To figure out which surface shader inputs are actually used in the various lighting passes, we’re generating small dummy pixel shaders, compile them with Cg and use Cg’s API to query used inputs & outputs. This way we can figure out, for example, that a normal map nor it’s texture coordinate is not actually used in Deferred Lighting final pass, and save some vertex shader instructions & a texcoord interpolator.

The code that is ultimately generated is compiled with various shader compilers depending on the target platform (Cg for Windows/Mac, XDK HLSL for Xbox 360, PS3 Cg for PS3, and our own fork of HLSL2GLSL for iPhone, Android and upcoming NativeClient port of Unity).

So yeah, that’s it. We’ll see where this goes next, or what happens when Unity 3 will be released. I hope more folks will try to write shaders!

Terrain Toolkit

The Terrain Toolkit is an integrated set of tools for the Unity Editor which is designed to streamline and improve the workflow involved in creating realistic terrains for games. The toolkit enables the creation of large scale, realistic and playable game worlds within a very short time span.

Unity already has a powerful terrain engine as one of its core features. The Terrain Toolkit expands on this by providing the user with the ability to rapidly create a variety of different landscapes directly within Unity without needing to use third party software. An additional advantage is that landscapes can now be generated dynamically at runtime. This means a Unity game can have a potentially infinite number of levels or landscapes with a negligible impact on the file size.

Integrated terrain creation tools

The Terrain Toolkit provides an intuitive interface within the Unity Editor allowing a Unity artist to simply generate, erode and texture terrain objects. Quick and easy-to-use presets are provided for first time users, while advanced users can tweak low level settings such as rainfall and evaporation to get exactly the result they are after.

Watch the tutorial video to learn more about using the Terrain Toolkit:

A powerful runtime API

The Terrain Toolkit comes with a powerful API that allows developers to call any of the terrain generation, erosion and texturing scripts at runtime.

This Unity web player shows the Terrain Toolkit API in action: http://www.sixtimesnothing.com/terraintoolkit/APIExample.html

Where to next?

I plan to continue work on the Terrain Toolkit by both making further improvements to existing features, as well as expanding on the feature set to allow even more flexibility in the creation of procedural terrain.

Here are some of the planned features for future versions of the Terrain Toolkit:

• Texture presets
• User defined presets
• The ability to apply pre-defined presets through the API
• Realtime destructible terrain
• The ability to create multiple, maskable terrain ‘layers’
• An advanced hydraulic erosion filter based on a shallow water model
• Glacial erosion

Documentation

Comprehensive documentation for the Terrain Toolkit and the API is included in HTML format. The latest version of the documentation can also be viewed here: http://www.sixtimesnothing.com/terraintoolkit

Questions and support

If you have any questions about the Terrain Toolkit, the API, or any suggestions for improvements or new features, please feel free to contact me on the Unity forums (Nekharoth) or through my developer blog: http://sixtimesnothing.wordpress.com

Get the Terrain Toolkit from our Resources section.

Well ok, it all started a bit before:

I exchanged some emails with David and Joachim and they invited me for a gamejam in their office. Then one thing led to another, I was young and needed money (oops! wrong topic) and on January 2006 I started working on this thing called “Unity”.

Unity was at version 1.2.1 then. Since then we’ve released about a dozen new versions, added hundreds (or thousands?) of new features, a handful of new platforms and have grown a lot.

Also, we stopped saying “Sales are INSANE!!!!11″ whenever they exceeded a whopping ten thousand euros per week. Seriously, that much money in 2006 was a big thing. Our Windows build machine was a single core Celeron with 512MB RAM because that’s what we could afford! Well ok, we’re still saying “sales are insane!” from time to time, just the threshold has gone way up.

Occasionally we’d get excited about strangest things. I think this email is about some car model from ATI that was on front page of our website in 2006. It’s beyond me why we’d put a car on Unity website, but somehow it seemed to make sense at the time.

It would take too much space to list all the awesome things that happened in those four years. I got to work on some things too, like Windows Web Player, Direct3D renderer, shadows, editor for Windows and whatnot. But I mostly concentrate on creating trouble, which does not seem to hinder Unity that much. I need to get more efficient!

Seriously though, it has been an amazing ride so far, and I hope it will only become better. Thanks to everyone at Unity Technologies and the community!

Rock on!

We want to try one more thing… On Friday work on something cool.

I think there are a lot of low hanging fruits in Unity where somebody with drive can just do something cool that pushes us forward. Things that are hard to put in words, but just make sense when you see it done.

So every Friday, developers can work on something cool, something they have been craving to do for a long time. A feature, a demo, a tutorial, a video tutorial, docs, cleaning the toilet, something on the website. Whatever. Be creative, push the envelope.

In other words, we started to work on “anything cool” on Fridays. This was dubbed as FAFF (Fridays Are For Fun).

Boy, what a positive effect on team morale that was!

It also resulted in some FAFF work being polished & promoted into official releases already. Things like:

• Animation Events in Unity 2.6 animation editor
• Display realtime brush preview in terrain editing tools
• Timeline Charts for Unity 2.6 profiler
• Custom Icons for standalone game builds
• Head Look Controller sample project
• Precise raycast-based snapping in Unity 2.6

…all started out as Friday projects!

Developers have experimented with lots of other stuff on Fridays. Some of them were just playing around or learning Unity better. Some Fridays were for evaluating various middleware packages, implementing some features or doing some fancy R&D. Some of the projects were an interesting kind of “fun”. Like improving internal build farm setup, a system for better tracking of crash reports, improving the website, or other similar things that often involve Perl scripts. Well, eveyone has his own kind of fun

So yeah. On Fridays, we just ignore the schedule and work on something cool. Something that pushes our own tech – something that we can’t defend from a business standpoint but that’s just too great to not get done. If it rocks, we’ll ship it. Because, at the end of the day – who knows where the next killer idea will come from.

The objective of this Unity Summer of Code project was to develop a toolset for the Unity Editor which would streamline and improve the workflow involved in creating terrains for games.

While there are currently a number of third party tools that can already produce high quality terrains, I felt that they lacked two vital qualities. Besides the impact of the additional cost of these tools – particularly to indie developers, when using these tools it is left to the artist to develop an effective workflow for getting height and texture maps out of their proprietary software and into Unity. This can be a somewhat painful process. Secondly and more importantly, it is impossible to leverage the power of the tools to procedurally generate landscapes at runtime. My aim was to overcome these constraints by integrating terrain generation tools directly into the Unity engine and editor.

The toolset I am developing is divided into three areas – terrain generation, erosion and texturing. So far there are 5 different erosion filters – thermal, tidal, wind and two types of hydraulic erosion, 2 ‘generators’ that use Voronoi and cloud fractal algorithms to create base terrain, a couple of useful tools that allow the user to smooth or normalise the terrain and finally a procedural terrain texturing tool.

All ten of these features will be provided both as filters which can be applied in the Unity Editor and as fully documented API functions which will allow users to generate terrain and apply filters at runtime through their own scripts. Additionally, thermal, fast-hydraulic and tidal erosion filters can also be applied directly to the terrain object using brushes in the editor.

Where it’s up to, what still needs to be done

I feel that the project has progressed well over the last five or so weeks. There have certainly been some obstacles and not everything has gone my way, but with just over a week remaining I am on track towards delivering at least 16 of the 19 features that I originally scoped for the project, plus two completely new ones. Given that my aim was to deliver a minimum of 12 of the scoped features within the Unity Summer of Code timeline, I’m quite happy with the result.

Over the next week my main aim is to tidy up the documentation for this project, which I believe is vital in allowing Unity artists and developers to get the most out of these tools. I will also focus on cleaning up my code and optimising it where possible and finally I will continue working away on the remaining features – namely a few UI tweaks, yet a third different approach to hydraulic erosion, and glacial erosion.

What went well

I started the project with a fairly limited knowledge about creating properly integrated custom tools for the Unity Editor and had only given the Editor GUI documentation a cursory glance. I found that despite a few minor hurdles, I was able to quite easily create a useful and intuitive interface for my tools. I was also pretty happy with some of the optimisations I was able made to my erosion code. Being able to run 50 thermal erosion iterations on a 512 x 512 terrain object in barely more than 6 seconds totally exceeded my wildest expectations.

Also, after many unsuccessful attempts to get Unity’s undo feature to work with my code, one of the Unity guys, Rune, was kind enough to explain it to us – and in typical Unity style, it turned out to be much simpler that I had thought it would be.

Even though I had originally considered undo as a ‘nice to have’ feature, I quickly realised that it was really an imperative. Getting the best results from these tools can require a bit of trial and error, and allowing users to run a filter, undo it, tweak a few settings and run it again without completely ruining their terrain object in the process is vital in making this project a practical solution for Unity artists and developers.

What didn’t go so well

While I consider my scripting skills in JavaScript and C# to be reasonably good, I have to admit that mathematics is not really my strong point. Some of the equations I used in my tools, such as Voronoi diagrams and the Diamond-Square algorithm were fairly easy to implement in code – but for the vast majority of my tools I initially approached the algorithms logically rather than algorithmically.

For the most part this worked well and produced good results. However, when I started looking for a more physically correct approach to erosion caused by fluid mediums – namely water and ice – I realised that the complexity of the mathematics involved in computational fluid dynamics, such as Shallow Water Models, Navier-Stokes equations and so on, was far beyond my level of understanding and well beyond my capacity to learn in such a short period. This was rather disappointing, as it has meant that I have not yet found an effective solution for glacial erosion.

That said, there is still a lot more that I want to achieve with this toolset. After I deliver my project at the end of the Unity Summer of Code, I fully intend to revisit some of these prohibitively complex features once I have the time to do all the learning and research that will be required to get them to work the way that I want them to.

I talked about developing graphics technology for small games (PDF slides). Mostly on hardware statistics, GPU features, testing and stability.

ReJ talked about low level iPhone (pre-3GS) rendering details (PDF slides). On inner workings of iPhone’s GPU, OpenGL ES drivers, command buffers, VFP assembly and so on. Some awesome tech stuff in there!

If you’re going to watch some demos from Assembly 2009, make sure to see:

• Frameranger (1st place demo). Imagine that on a big screen and a good sound system!
• The Golden Path (3rd place demo) – for something fresh.
• Muon Baryon (1st place 4 kilobyte intro) – that’s what kids do with sphere marching on the GPU these days. Everything in 4 kilobytes!

…see you at Breakpoint next year?

Currently Unity mixes the fixed function pipeline for vertex lit objects and vertex shaders for pixel lit lights. On Direct3D this creates some rendering artifacts on very close or self intersecting surfaces. We want to end this by implementing a full fixed function emulator in vertex shaders. You won’t have to do anything, it will just work!

Well, the good news is that we have done it, and it is definitely coming in Unity 2.6!

It is funny, because the problem seems like a simple one: “there are double-lighting artifacts on close surfaces”. The solution, however, is “we need to implement whole fixed function pipeline manually”, which involves combining shader assembly fragments, doing register allocations, packing vertex pipeline state into bits and other low level hacking.

In other words, it required quite a bit of thinking and implementation effort, so I decided to write up a technical report on how we solved it. If you’re a graphics programmer or love programmer speak or just have nothing better to do: here’s the report.

Unity 2.5 was the first version of the editor running on Windows and eventually was released in March 2009. Some development of it started at end of 2007, just after Unity 2.0 was released. First there was some serious code shuffling, converting from Objective C into C++, refactoring into platform independent interfaces and similar highly invisible stuff.

Here’s the first Unity running on Windows screenshot I could find (this is February 2008):

This was the version dubbed “Can Haz Menus”. However, a couple of days later it was “CanHazGUI” already:

That’s true, whole editor at that time only had a view into the Project, and you could not really do anything with it, except to use the scrollbar. Or maybe collapse and expand the folders – I forget. Things were progressing really fast though. About a week later I sent this status report mail:

Working on Windows editor. Not much to say, so I’ll attach a cute picture instead. It’s getting there.

(Back then I was the only person with a working 2.5 for Windows build, and I did work separately from the main office. Occasionally I had to send screenshots as an argument for getting my salary!)

…and then came work for Unity 2.1, Unity iPhone, Fusion Fall and whatnot. Work on 2.5 was progressing, but at a bit slower pace. And of course, “getting something on screen” is much less work than “getting it to actually work”.

Here’s how the Windows editor looked in September 2008:

This was approaching final look of 2.5 already. Of course, occasionally we’d run into funny situations where it did not quite work. For example, at that time if you dropped a refractive glass shader into scene, it would turn the editor into this:

It took another half a year to add all new & missing features, fix a ton of large & small issues until it got to a real release:

Well, here they are: unity3d.com/webplayer/hwstats

Operating system versions, desktop resolutions, graphics driver versions, shader models, memory sizes and some more. All broken down by quarter so some sort of “trends” can be seen (sure, changes can be caused by general hardware change or simply different people groups playing different games).

Enjoy!

However, the “casual gamer”, which is what Unity games are mostly targeted at, probably has slightly different hardware. “Slightly” being a very relative term of course.

Lo and behold – we have a glimpse into that data.

How? First time the Unity Web Player is installed, it submits anonymous hardware details (details in the EULA). This happens only once, and contains no personally identifiable information. It’s much like visitor statistics trackers on the websites that gather your OS, browser information and whatnot.

We still haven’t prepared nicely laid out, colored and formatted reports into that data that we can share publicly like Valve, but we’re working on that. So in the meantime I’ll share some images without fancy graphs or colors, bear with me.

Remember, all this data is from people who installed Unity Web Player (most likely because they wanted to play some Unity content on the web). Hardware of standalone game players might be different, and hardware of your game’s players might be different as well. The data set is well over a million samples at the moment.

Enough talk, let’s show some pictures.

What operating systems do we have?

This one should be easy to understand.

What Windows versions are out there?

We should know OS X versions as well, right? Hey, Leopard already took over Tiger!

Hmm, looks like 64 bit Windows haven’t really taken off yet…

Enough about OS, what about CPU?

Poor Transmeta… (for the record, that’s 43 Transmeta CPUs)

Multicore CPUs are taking off (well, at least dual core ones).

Graphics card?

Capabilities of the graphics cards.

The capabilities are not as bad (certainly better than I expected). I mean, about 70% of them support shader model 2.0 or higher!

What is is troubling is the 3.7% that run in OpenGL 1.1 software mode… That means they don’t have graphics drivers installed or have disabled hardware acceleration. This is just crazy, I mean, without the drivers, even dragging windows around is horribly slow… how people even use the computer this way?

Another interesting point is that DirectX 8.0 level hardware (GeForce 3/4Ti) is pretty much dead. I guess that’s because there never was an entry-level graphics card from that range (before there was GeForce 4MX, which is DX7 level, and it was followed by GeForce FX 5200, which is DX9 level).

However, the above was technical capabilities of the graphics cards. Let’s take a look at which cards are out there:

Uh-oh. Can you say “low end”? The first decent card here is GeForce 8600, 15th on the list. Everything above – slow, slow, slow. Some horribly slow. Well, that’s casual gamer…

Here’s a rough performance indicator, graphics card fillrate in gigapixels/second:

For the record, fillrate of GeForce 8600 is 4.3 GP/s, Radeon X1600 is 2 GP/s, Radeon HD 2600 is 2.8 GP/s, and Intel 945 (GMA 950) is 1.6 GP/s. The difference from high-end to low-end in video memory bandwidth is even larger.

On a somewhat unrelated note, here’s DirectX 10 card distribution:

DirectX 10 can only be used starting with Windows Vista, and on DX10 capable graphics card. So that’s 2.6% of the Unity web game players – still not a very significant amount.

In conclusion

So here was the glimpse. We’ll prepare more and nicer reports into the hardware data sometime soon. Stay tuned!

Edit: it’s live – unity3d.com/webplayer/hwstats

Here’s a glimpse into what is needed to get it working.

The Plan, straight from our internal planning wiki, with some explanations added:

o) Convert all terrain code to use Intermediate Mode Render Calls. This makes all the points below possible, and makes things like Projectors or wireframe terrain rendering in scene view pretty much just work.

o) Update terrain shaders for proper rendering order and soft vegetation. Expand ShaderLab so that a shader pass can indicate it should only be executed when Soft Vegetation is on.

o) Proper vertex lights on terrain, implemented in a vertex shader. Currently terrain handles up to four directional lights; we need to expand it to handle point lights as well. I think we’ll treat spot lights (when in vertex lit mode) as point lights; as calculating multiple spot lights in a vertex shader is next to impossible because of instruction count limits. Spotlights are not very useful in vertex lighting mode, right?

o) Implemented pixel lit shaders for terrain. This will get light cookies, per-pixel lit terrain, and ability for terrain to receive shadows.

o) Add option somewhere for pixel or vertex lit terrain (defaults to vertex lit).

o) Proper vertex lights on trees. Very similar to “vertex lights on terrain” point above.

o) Make tree shaders able to cast shadows. Because trees are scaled & animated in a vertex shader, the shadow-casting shader needs to apply the same animation as well.

o) Make Projectors not affect grass / trees. How? We’ll expand ShaderLab so a any shader can say “no Projectors on me, thanks”, and modify grass & tree shaders to do that.

o) Make projectors not affect terrain when game content is built with Unity 2.0. This is web player backwards compatibility – there are Unity games out there that use projectors and terrains, and they should continue functioning exactly like they did before. So Projectors should not affect terrain in this case.

o) Make tree shadows not s*cklook bad when they are squashed for billboarding. Trees are squashed into a planar shape when they approach billboards; this is to make the mesh-to-billboard transition less noticeable. However, this makes shadows from those trees look very wrong. Not sure how to handle this; I think we’ll just stop casting shadows when trees are starting to squash.

o) Redo Lightmapped terrain shaders:

• Use existing shaders for game content built with Unity 2.0. Again, existing web player games should work like they did before! For reference, in Unity 2.0.2 lightmapped terrain was not affected by any lights at all.
• Make new set of terrain shaders that follows the convention of other Lightmapped shaders: lightmap, plus additional lights on top. Whoa, hey, we need two new sets of those shaders: for vertex-lit terrain and for pixel-lit terrain!

o) Optimize the bejesus out of rendering internals. Why – because the old terrain rendering code was like this:

1. Figure out which parts of terrain and vegetation are visible.
2. Set terrain shader.
3. For each terrain chunk: draw it’s mesh!
4. Set tree shader.
5. For each tree: draw it’s mesh!
6. …similar for billboarded trees, grass, and so on.

This is all simple & nice. Of course this was exactly the problem why shadows, projectors and such didn’t work on terrain – everything was just rendered immediately. Now, the new terrain code just submits any visible parts of terrain as Intermediate Renderers, and the rendering code figures the rest out… But that means it’s going through generic rendering “pipeline”, that has to figure out lighting, sort out objects, set shaders and their parameters, etc. The major overhead in switching to this “proper rendering” is the lost ability to do this:

1. Set the shader.
2. Draw lots and lots of objects with it.

The rendering code in Unity does sort objects by shader, and does ultimately figure out that they are using same shader, possibly same set of textures, even possibly same set of lights and same set of material parameters – but some time is spent just figuring out that nothing actually changed between the objects.

Well, the upside is that optimizing rendering internals makes everything else run somewhat faster as well!

(I am here right now… the points above are done, the points below are on the list)

o) Make hills/trees outside of view be able to cast shadows (how?)

o) Expand shaders of detail meshes so they can cast shadows.

o) Optimize terrain & tree vertex lit shaders. Right now they process four vertex lights, where each can be directional or point light. This is noticeably slower than old shaders, that only supported directional lights (point lights need more calculations). We need to write multiple combinations of those shaders, specialized for specific situations (“one directional light”, “two direcitonal lights”, “two arbitrary lights”, and so on).

o) Add option somewhere for terrain to not cast shadows.

o) Add option somewhere for trees to not cast shadows.

o) Add option somewhere for detail objects to not cast shadows.

o) (maybe, don’t know how to do) Option for grass to receive shadows.

o) Reduce memory allocations in terrain code. Because of intermediate rendering, some of temporary arrays are not actually needed anymore.

Whew, that was quite a list! I’m not sure I’m able to even read through all of that… well, back to work.

…
Joachim: yeah it’s awesome
Joachim: function Start () { /* super secret code snippet */ }
Aras: whoa
Joachim: it rocks
Joachim: automatically instantiates material for you
Joachim: the same way we do it for scripts
Joachim: best thing since sliced bread i think
Aras:
Aras: hey, back in the day the terrain was best thing since sliced bread!
Joachim: sliced bread appears to be always worse depending on what i work on.
Aras: maybe sliced bread is just not so good in fact
Aras: sliced bread is overrated

And no, I won’t say what we were talking about

Wait a moment… Unity plugin makes nice wobbling web page elements not wobble anymore!? Sounds like an interesting issue…

So I prepared for a debug session and tried the usual “divide by two until you locate the problem” approach.

• Unity Web Player is composed of two parts: a small browser plugin, and the actual “engine” (let’s call it “runtime”). First I change the plugin so that it only loads the data, but never loads or starts the runtime. Everything works. So the problem is not in the plugin. Good.
• Load the runtime and do basic initialization (create child window, load Mono, …), but never actually start playing the content – everything works.
• Load the runtime and fully initialize everything, but never actually start playing the content – the bug appears! By now I know that the problem is somewhere in the initialization.

Initialization reads some settings from the data file, creates some “manager objects” for the runtime, initializes graphics device, loads first game “level” and then the game can play.

What of the above could cause something inside browser’s JavaScript engine stop working? And do that only on Windows, and only on some browsers? My first guess was the most platform-specific part: intialization of the graphics device, which on Windows usually happens to be Direct3D.

So I continued:

• Try using OpenGL instead of Direct3D – everything works. By now it’s confirmed that initializing Direct3D causes something else in the browser not work.
• “A-ha!” moment: tell Direct3D to not change floating point precision (via a create flag). Voilà, everything works!

I don’t know how I actually came up with the idea of testing floating point precision flag. Maybe I remembered some related problems we had a while ago, where Direct3D would cause timing calculations be “off”, if the user’s machine was not rebooted for a couple of weeks or more. That time around we properly changed our timing code to use 64 bit integers, but left Direct3D precision setting intact.

Side note: Intel x86 floating point unit (FPU) can operate in various precision modes, usually 32, 64 or 80 bit. By default Direct3D 9 sets FPU precision to 32 bit (i.e. single precision). Telling D3D to not change FPU settings could lower performance somewhat, but in my tests it did not have any noticeable impact.

So there it was. A debugging session, one line of change in the code, and fancy javascript webpage animations work on Windows in Firefox and Safari. This is coming out in Unity 2.0.2 update soon.

The moral? Something in one place can affect seemingly completely unrelated things in another place!

Draw( position, rotation, mesh, material );

And that would make the mesh appear with the given material at the given position on the screen, and it would just work with pixel lights, shadows, projectors and whatnot.

A step aside on why this is needed: in Unity 2.0 the terrain engine does not fully integrate with pixel lights or shadows; and Projectors don’t work with it either. The reason is that terrain is rendered immediately: at some point terrains gets it’s render call, draws the chunks of the terrain, the trees, the detail objects and all of this goes straight to the screen.

For shadows, pixel lights or projectors to work, however, something has to be remembered, so that the most important lights can be calculated for each object, so that shadow casters could be rendered, and so on.

With intermediate mode render calls the terrain engine could “submit” this lightweight rendering information: mesh, material, position; and the rendering system would figure out everything else at rendering time.

Ok, the above sounds like a plan. What goes into developing such feature?

The basic functionality is obvious. Draw() above would not actually draw anything immediately, just merely record the information somewhere in the scene: “this mesh with this material will have to be drawn here”. At the end of the frame, all this information would be discarded, so Draw() persists for one frame only.

Some design issues immediately pop up:

• Will this mesh be rendered in all cameras, or just single one? (which one?)
• Maybe it’s drawn in all cameras if called from inside Update(), and drawn for current camera if called from inside OnPreCull() or OnPreRender()?
• Or maybe we add an explicit “camera” parameter; and the mesh will be rendered in all cameras if no camera is indicated?
• What to do if Draw() is called from inside of Start(), Awake() and other functions that are not related to “frame” or “camera”?

Some of the work required is also obvious: Right now all renderable things in Unity are Renderer components. That means they live in some game object, and so on. For intermediate mode calls, we need to put base rendering functionality somewhere “up” – so that a renderable thing can not necessarily be a Component in a game object.

• Make a leaner base renderer class that does not derive from Component (call it BaseRenderer). Move out base functionality from Renderer to BaseRenderer as needed. Make Renderer be both a Component and BaseRenderer.
• Make the rendering code (that figures out lights, does shadows, …) operate on BaseRenderers, not Renderers.
• Make IntermediateRenderer that would be used for intermediate mode. It has transformation matrix, pointers to mesh and material. Make it setup and update all related internal structures that describe Unity scene.
  • Possibly need distinction between scene-wide meshes and camera-specific meshes.
  • Clear all intermediate renderers at end of frame. And make this work in the editor with multiple views into the scene present.

With the above, we can go and write some code to implement the feature. Look ma, no game objects are here, just two lines of code drawing 100 meshes with random rotation in a loop! And it works with shadows!

So the proof that the base feature works is there already (we say it’s a killer feature for making demoscene demos). We still have the design issues of the above, and some more that popped up while writing the code:

• Is culling performed on these meshes or not? It’s useful if the mesh will be rendered for all cameras, and potentially redundant if the mesh is drawn for single camera only.
• Are layer masks indicated anywhere? Add a parameter to the function?
• What if for each mesh rendered I want to change something in the material? (change some value for a shader, use a different texture, different tint color, …) Should I instantiate materials myself? Should the API instantiate materials internally? Should I supply variable number of arguments saying “and do these changes to the material before drawing”?

It’s a long rocky road between thinking up a feature and shipping it. Surprisingly, actual development does not take up much time; the most difficult part is answering all the design questions.

Anyway, after all design issues are solved (or… ignored!), the feature usually is shipped. Yay!