Advanced Shading and Lighting for Mobile

June 8, 2011 in Technology

 

Hello everyone. You have probably already heard about Shadowgun, the new mobile game developed by MADFINGER Games. You might also have seen Shadowgun being demoed at WWDC on stage during the Graphics State of the Union talk.

Needless to say, we’re very proud of the next great creation made in Unity, and we love how it delivers console quality to the mobile with the help of Unity’s awesome tech. It runs at an impressive 60fps on the iPad 2 and 30fps on the iPad, iPhone 4 & 3GS. It will also be released on Android. Since lighting is one of the main elements making the game look so good, here are some notes about the techniques we used.

Character shading

All characters in the game use the same per pixel normal mapped isotropic BRDF pixel shader, which goes beyond the usual Phong shading. It gives a great effect on the skin of the character, metal and cloth. Our solution combines off-line precomputed texture which describes how surface interacts with lighting and run-time per-pixel calculations to achieve both artistic control and superb performance.

Light probes

shadowgun-greenLight probes are an advanced technique for lighting dynamic objects and characters with high-quality baked lighting. It is possible to capture incoming direct and bounced lighting within a scene at a number of points – light probes. As a character moves through the scene, nearby probes are picked, interpolated, and the result is then passed to the shader to efficiently light the surface. There it can be sampled per-vertex or per-pixel, if you want to use the normalmap-perturbed normal to do so.

Choosing and interpolating the right probes (and doing it fast!) is a hard problem, though. We have developed a technique, which takes an arbitrary set of probes and performs Delaunay tetrahedralization on them to divide the entire space into tetrahedron-shaped cells.

Shadowgun_muzzleThat gives us fast interpolation with great properties and no light popping – when you’re in a given tetrahedron, weights for the four corner probes are your barycentric coordinates! Finding the right tetrahedron can also be done efficiently by checking the one from last frame – if it’s not a hit (most of the time it is!), we check it’s neighbours. Finally we make sure interpolation works outside of the volume of the probes just as fast and as good with a clever projection onto the hull surface.

I’m especially proud of the light probes, since we managed to shape them up to be extremely fast and have low memory footprint – perfect even on mobiles and giving so much bang for the buck. As far as we know, it’s also the first time it has been used to such extent (at all?) on mobile devices.

Muzzle flashes

As a bonus, since we’re already rendering every character with a light probe, we can project any additional lights onto the probe as well. We do that for the muzzle flash effect. For a very low CPU cost and no GPU cost characters get lit by dynamic light.

God rays, light shafts, light flares and fog

God rays, light shafts, light flares and fog are rendered using both static and animated alpha-blended geometry with very optimized vertex and pixel shaders to save fillrate.

Lightmaps

Lightmaps were baked with one click in Unity using Beast to capture all the beautiful Global Illumination on static geometry. To give the objects a more detailed look while still keeping the shaders extremely fast, we calculate approximate specular reflections per vertex and then modulate them per pixel with a gloss mask to differentiate between shiny and rough parts. Reflective objects additionally use a pre-baked cube map matching the surroundings.

Caustics and character shadows

Caustics from the light passing through the green fluid, character shadows and rotating ventilator shadows were simulated with textures projected onto tight geometry (to keep the fillrate requirements low) and then animated in the shaders.

shadowgun-in-game-unity

We plan to release light probes with Unity 3.5; all the other effects can already be achieved in Unity! :)

Comments (54)

Subscribe to comments
  1. rahmat

    August 20, 2011 at 4:04 am / 

    gokillllllllllllllllllllllllll

  2. RHD

    July 29, 2011 at 11:44 am / 

    OK, have found some of the rewritten shaders, but would instructions kill you? How do I get my boat floating again and my underwater effect back? Never mind the God rays. Thanks though for some splendid stuff.

  3. RHD

    July 29, 2011 at 10:39 am / 

    I hate to sound ungrateful but I’ve just installed Unity 3.4 and now the OTHER half of my shaders are broken.

    It would be good if Unity could provide us with rewritten ones to replace the broken ones as quite a lot of people have no clue as to how to code Shaders. Including me.
    Thanks

  4. WillBellJr

    June 30, 2011 at 1:35 pm / 

    All these changes really sound great!

    I’m still hoping for whatever route you take with the GUI improvements, I’ll be able to do this:
    http://forum.unity3d.com/threads/23336-Better-2D-Graphics-Support

  5. Rodrigo

    June 29, 2011 at 5:50 pm / 

    uOww!

  6. Aj

    June 29, 2011 at 12:08 pm / 

    Unity Smashed it AGAIN! It seems like Torque and Unity are really Striving to Be the BEST!

  7. Aras Pranckevičius

    June 28, 2011 at 6:42 am / 

    @Jason: seriously, if you really think that Unity’s features or the roadmap is “simply unacceptable”… why not use something else? We try to work on various features as hard as we can, and to prioritize them as good as we can. If all our work in simply unacceptable for you, then oh well, maybe Unity is just not for you. Good luck!

  8. Jason Amstrad

    June 28, 2011 at 6:01 am / 

    @Aras
    First no pathfinding until 3.5 and now the promised dynamic fluid surfaces are not on the roadmap at all. I’m sorry but this is simply unacceptable. Cheaper engines like UDK have more features than this

  9. Matt

    June 11, 2011 at 8:09 am / 

    Very impressive stuff!

    It’s going to make the wait for the updates to 3.4 and 3.5 even harder now though!! :)

  10. Nathali Abbortini

    June 10, 2011 at 9:11 am / 

    @Robert Cupisz
    Nice to hear that it runs faster than UDK.
    You guys are the best.
    Thanks Unity Tech.

  11. Robert Cupisz

    June 10, 2011 at 5:57 am / 

    @Nathali: the basic concept of sampling the irradiance in space and then using the samples to light the dynamic objects is similar. But the execution is completely different.

    It uses dense uniform grids within a bunch of bounding boxes placed around the scene. I don’t know how it samples the data, but my best guess is that it finds all the probes within a given range and assigns weights to each one – or discards it if a raycast fails. If that’s the case, it suffers from the resulting interpolated probe being very blurred and failing to show contrasts in the lighting, even though they were correctly captured by the baked probes. That kind of interpolation is slow and a regular grid takes up a ton of memory.

    On the other hand our solution gives you probably the best possible interpolation quality, is fast, has a low memory footprint and runs on mobiles with no problem :)

  12. Nathali Abbortini

    June 10, 2011 at 5:24 am / 

    Are “light probes” the same thing as what the unreal engine calls “light environments” ?

  13. pixnlove

    June 10, 2011 at 4:55 am / 

    At the moment I am unable to use NormalMap with LighMapping.
    Normal Map just do not show when Lightmap are unable.
    Does that mean we will be able to do that soon?

    You Unity3D guy are too good to be true, carry one the Great work.
    You madfinger guys are too talented I wish I would be able to design
    a game that look like, that is pure awesoness!

  14. Aras Pranckevičius

    June 9, 2011 at 9:45 pm / 

    @Manon Seppen: we don’t know, so the best answer would be “maybe someday”. Also, if you have “instanced draw calls” in mind, you’d have to know they don’t work (or don’t bring any performance benefits) on all platforms. On PC with Direct3D – sure. On consoles? Not so much (draw calls are really cheap there). On mobiles? They don’t exist.

  15. Holland

    June 9, 2011 at 6:40 pm / 

    Did I just saw penis on the statue? How are they going to get pass the Apple censors?

  16. Robert Cupisz

    June 9, 2011 at 3:24 pm / 

    @Damien: Light probes will be available across all platforms of course.

    @Baronium, @Laurent: I’m excited to see what tools the community will come up with to allow artists to paint lighting into the probes or other funky stuff like that :)

  17. PedroAfonso

    June 9, 2011 at 2:48 pm / 

    it will probably be!

  18. Damien

    June 9, 2011 at 2:28 pm / 

    Will this light probes feature be just for mobile platforms, or will it included into Unity for desktops and web apps as well?

    Please say it will be … pretty please!

  19. Curious

    June 9, 2011 at 11:07 am / 

    Hmm, this is great :D
    Can’t wait for 3.5 though ;)

    We’re waiting for the preview of 3.4 guys ;)

  20. PeterB

    June 9, 2011 at 7:35 am / 

    Hm. There’s at least one package in the Unity Store which requires Unity3d v3.4.

  21. Edinson

    June 9, 2011 at 7:07 am / 

    This is awesome, good bye UDK!

  22. Baronium

    June 9, 2011 at 5:01 am / 

    @Dirk41
    Aras said soon. This is soon enough.

    I’m excited what opportunities we have with those light probes in the future.

  23. Dirk41

    June 9, 2011 at 4:45 am / 

    @Aras Pranckevičius
    How soon ?

  24. Aras Pranckevičius

    June 9, 2011 at 2:38 am / 

    @PedroAfonso, watch our blog. Soon.

  25. PedroAfonso

    June 9, 2011 at 2:23 am / 

    @Aras, what features will there be in 3.4?

    Since UT is so close to release it, could you give us a preview of some features it will bring?

  26. Aras Pranckevičius

    June 9, 2011 at 1:35 am / 

    @Greg: yes. And since Unity already applies spherical harmonics lighting in forward rendering (all “not important” lights + ambient goes there), taking light probes into account has no additional cost at all on the GPU. There’s only very small cost on the CPU per object (to find, interpolate the probes and add them to SH info).

  27. Greg

    June 9, 2011 at 1:17 am / 

    I am curious, are the light probes modeled using spherical harmonics ?

  28. Aras Pranckevičius

    June 8, 2011 at 10:50 pm / 

    Oh, and everyone asking about Unity 3.4 vs 3.5: we’re at the finishing stages of Unity 3.4 release… we aren’t skipping it! Light Probes will come into the release after that (3.5).

  29. Ashkan

    June 8, 2011 at 10:37 pm / 

    awesome skin shader.
    lightprobes are really a great feature. waiting for features is something natural, people who they are working on it are humans like us believe me!!!
    they need time to do their jobs, they are not God or super fast robots who could do coding!!! :)

  30. hippocoder

    June 8, 2011 at 9:03 pm / 

    Wow what can I say. I think this looks fantastic, and by far the best looking game on the asset store. at least 50% of the result is great art though. The shaders would be nice for unity pro users, an incentive to upgrade. Fantastic work, certainly better looking than infinity blade but thats a personal observation.

    Finally, I’m very tired of children posting comments into blogs or forums.

  31. Fractal Core

    June 8, 2011 at 8:20 pm / 

    Sorry about that Giyomu, I got the name wrong. I meant Hanzo is the lame-ass “PLAYER”

  32. Fractal Core

    June 8, 2011 at 8:18 pm / 

    Wow, Giyomu doesn’t get it, at all. This is a demonstration of the technology being used by the game. And by your own admission you don’t even WANT to make your own game. What the hell are you doing on a unity blog leaving a REVIEW of a game you haven’t played.

    We all know the brown-and-grey-type-of-game-is-unoriginal argument… Boring. Try posting what you think of how well Unity is matching something done in UDK.

    And before mentioning Infinity Blade, go find out which techniques they’re using inside UDK, then come back and find out what Unity can do. It comes down to what the artists can do with the technology, and something very close to Infinity Blade could easily be done with Unity. Not that you’d care, since you’re JUST a “PLAYER”

  33. Giyomu

    June 8, 2011 at 4:10 pm / 

    @Hanzo, what the point of this stupid reflexion ? You are free to make your own original no bland game with no lame art :)

    and for UT guys well as usual nice improvement on the engine, the mobile tech become really powerful, well worth wait for 3.5 imho.

  34. Hanzo

    June 8, 2011 at 3:53 pm / 

    Come on!
    That looks like a Gears of War or Bulletstorm clone! Do we really need more bland and unoriginal games with lame art?

  35. ZroeX

    June 8, 2011 at 3:17 pm / 

    Wow!!
    How amazing!!
    The next update to 3.5 directly?

    By the way,the game demo is look like “Gears of War”.
    ^__^

  36. Julio Sanchez

    June 8, 2011 at 2:27 pm / 

    What do you mean by “animated in the shaders”? when you talk about the rotating ventilator shadows.

    Best Regards

  37. Aras Pranckevičius

    June 8, 2011 at 11:45 am / 

    @Dev.D:
    1) the exact shader used in this game would be rendered in forward path. Most of interesting lighting calculation in deferred is done by a single shader (thus it’s hard to make really different materials there), but you can surely override the shader that’s used in Unity by default. For example, replace the current Blinn-Phong with a properly energy preserving one, etc.
    2) Well, a much less fancy sounding name for that texture would be “a lighting lookup texture indexed by (N.L, N.H)”. Turns out doing an extra texel fetch is faster on current mobiles than computing pow() + couple more math instructions. And it allows more flexibility than standard Blinn Phong BRDF. It’s still limited to isotropic ones, and the ones that can be indexed by (N.L, N.H) of course. The concept is explained in this article.

    @Jason Amstrad: you know, I was thinking “hey we do a development related blog post, Jason will come and ask for pathfinding”. I was right! ;) Yes, we know you want pathfinding, and we are working on it. This blog post is very much not about pathfinding though. Can we keep it on topic please?

    @Jon: my guess on the “very optimized shaders” – well, shaders that do almost nothing. For example (just guessing), maybe some of those “special fx planes” don’t even use textures. The vertex shader would then only do one matrix multiply to transform the vertex position, and the pixel shader would only return a color. To save additional fillrate you can employ other tricks, for example when camera gets close to some “god ray plane” object (so it would cover large area of the screen) – scale it down and fade it’s color our. This is not “correct” (objects shouldn’t become smaller when getting closer to them), but saves a ton of fillrate!

  38. Jon

    June 8, 2011 at 8:35 am / 

    “God rays, light shafts, light flares and fog are rendered using … with very optimized vertex and pixel shaders to save fillrate.”

    Any chance we can get a look at that shader? I am really curious how it solves/helps the fillrate seeing as how the retina and iPad are usually fillrate limited.

  39. Dirk41

    June 8, 2011 at 8:26 am / 

    @Jason Amstrad

    what does material instancing mean ?

  40. Manon Seppen

    June 8, 2011 at 8:24 am / 

    It would be nice if unity also supported material instancing.
    And also dynamic fluid surfaces and wave generation,if a character moves through the surface for example.

  41. Jason Amstrad

    June 8, 2011 at 8:22 am / 

    Unity was working on an internal pathfinding solution if I’m not mistaken.
    How about some news on that ?

  42. JoeW

    June 8, 2011 at 8:08 am / 

    Ah – suck. Lightprobes have to wait until 3.5? Oh well, all good things come to those who wait, eh? I’ve still got a ways to go before I’m ready to implement them anyway…. sigh.

    Nice work, guys – I love the fact that Unity is being so aggressively developed, and I certainly appreciate all the hard work.

  43. Jason Amstrad

    June 8, 2011 at 7:09 am / 

    so what will be new for unity 3.4 then ?

  44. Juan Sebastian munoz

    June 8, 2011 at 6:47 am / 

    Wow!,

    Awesome job Robert & Kuba! :D

  45. Dev.D

    June 8, 2011 at 5:58 am / 

    A couple of question:
    1. does the BRDF simulation work equally well on the deferred rendering path as well?

    2. An specific requirements for the LUTs texture formats? I’m hope since this is actually working (great) on the mobile – you guys aren’t using anything exotic.

    Overall excellent news :) . I’m glad we are getting a tweak-able BRDF system (shader & perhaps scripts) embedded into Unity. And am glad you are doing it for us . This is great news indeed as a single BRDF material can encapsulate multiple physical materials :). Our graphics guys used a similar system on APB for our customized character reducing the need for multiple materials per character to one . Great for things like fur , leather , skin esp when they appear on the same character … exciting news!! ^__^.

    -Dev

  46. Toha Yamaji

    June 8, 2011 at 5:31 am / 

    Now we have a link to paste every new forum topic about “Unity vs UDK: graphics”.

  47. thienhaflash

    June 8, 2011 at 5:23 am / 

    Very impressive ! I really like the light probe idea ! Unity3D is shaping 3D’s future, yay !

  48. rean

    June 8, 2011 at 3:13 am / 

    3.5….!!!???? ow come on guys T_T……. how about 3.4 :p

  49. Martijn Dekker

    June 8, 2011 at 2:58 am / 

    Very nice work! love to see some rocking graphics inside unity!

  50. Aras Pranckevičius

    June 8, 2011 at 2:22 am / 

    @TJHeuvel: yes, piece of cake! Actually we plan to put the editor scripts to generate lookup textures & the shader itself onto Asset Store for free soon. Just didn’t get around to actually do it yet (everyone trying to ship Unity 3.4, or something…)

  51. TJHeuvel

    June 8, 2011 at 2:12 am / 

    So all we need to do is write a per pixel normal mapped isotropic bidirectional reflectance distribution function pixel shader? Sounds easy!

  52. Laurent

    June 8, 2011 at 1:36 am / 

    That is so impressive.
    I see you solved the tetrahedra interpolation quite nicely :)

    And I just saw your answer in the old thread on probes. Great to hear you are exposing enough for probe painting and implementing probe prefab.
    On 3. “setting intensity of probe to a minus value to create shadow” I didn’t specify I wasn’t talking about cast shadows but shade, as in the darkness within a cave that comes from occlusion from the sky and sun lightsources. We used a trick back in the days of softimage 3.6, before we had ambient occlusion, where we’d place a negative light inside a cave to darken everything in it. Maybe it’s not possible for the same reason we can’t have negative lights – compatibility with the old ATI.

    Anyway thanks for creating the tools for cheap artistic lighting !!!

  53. JinWanDaLaoHu

    June 8, 2011 at 1:21 am / 

    What’s the BRDF pixel shader,is it in the Unity default shader?

  54. Quick Fingers

    June 8, 2011 at 12:58 am / 

    :O Very nice! 3.5 though! 3.4 isn’t out yet and your already tormenting us with things we will have to wait longer for :)

Comments are closed.