Introducing Sherman (Part 1) – a Unity project featuring Real time fur, HDRP and Visual FX Graph for animators
Created by the Emmy-winning team that brought you Baymax Dreams, Sherman is a new real time Unity short that delivers the most advanced real time fur ever!
My name is Mike Wuetherick, and I am the Head of Tech for Unity’s Innovation Group. Shortly after joining Unity 3 years ago, I helped found an Innovation team dedicated to pushing Unity’s capabilities for CG Animation and Film. Since then, my team has had the pleasure of working on collaborations with Neill Blomkamp’s Oats Studios (Adam Episode 2 & 3), Neth Nom on Sonder, and most recently the Baymax Dreams shorts with Disney Television & Animation.
This is part 1 of our Blog series on Sherman, where we will cover the project background, techniques used for blocking out the short, lookdev and camera layout. Be sure to check out Part 2, where we will discuss our use of Alembic for the character animations, Visual FX Graph and details about the Fur implementation!
Table of Contents
1. Project Background
2. Getting Started – the Tools
3. On the shoulders of Giants
4. Blocking out the Short
5. Lookdev with HDRP
6. Camera Layout with Cinemachine
After the Baymax Dreams project wrapped, we had the opportunity to continue working with a few of the key people that brought the Baymax shorts to life, including the Lead Animator, Bryan Larson on a new short.
The team on the production included Mark Droste (Baymax Dreams assistant director), Steven Shmuely (character modeler for Baymax Dreams), as well as our core team of John Parsaie (Graphics Engineer), Jean-Philippe Leroux (Lighting supervisor) and Adam Myhill (Cinemachine creator and Creative Director).
Leading the initial creative for the project was Bryan Larson, the lead animator for Baymax Dreams, and Nicolas Langois-Demers handled the character rigging. We were lucky to bring Mark & Steven onto the team full time after Baymax completed to help us push Unity’s capabilities with projects like this. On the production side, Andy Wood (Baymax Dreams’ line producer) managed the production, with Isabelle Riva leading the team and serving as Executive Producer. To round out the team, we were very fortunate to borrow the amazing talents of Vlad Neykov for the work with the new Visual Effect Graph (which our team had not used previously).
The core of the short was produced by a very small core team of roughly 8 from roughly mid-September through mid-December, after which point we had locked animation. The rest of the production was focused on R&D for the Fur among other things.
I’d also like to thank everyone at Unity that helped out during the creation of Sherman, in particular the Film R&D and EDU teams. Collaborations on projects like Sherman help us push what the platform is capable of, and wouldn’t be possible without their hard work and support.
With that said, I’m proud to introduce you to our latest short, Sherman!
Getting Started – the Tools
During the Baymax Dreams production, the team had become very familiar with the new HDRP renderer in Unity, including submitting a few key changes back to HDRP for improved shadow filtering & PCSS. For the new project, we wanted to continue to push the renderer and toolset as far as possible. Our graphics engineer, John Parsaie took the lead, and his work has really allowed us to raise the bar in many areas, most notably, fur rendering for real time.
For the Sherman production, we used Unity 2018.3 (and now the 2018.4 LTS version), along with a number of Unity packages, including Timeline, Cinemachine, Recorder, Alembic, FBX Exporter, Visual Effect Graph, Post Processing, HDRP among others.
During the creation of the short, we developed a number of tools & utilities, some of which we are proud to be able to share as a part of the Film / TV Toolbox package that is available from Unity’s Package Manager. The rest are in our ‘film library’ that we are in the process of migrating to the toolbox.
On the shoulders of Giants
As with most things, the work that the team did on Sherman was not in isolation. There is one project in particular that is worth mentioning – Marza Animation Planet’s short film ‘The Gift’.
Released in 2016, The Gift was created in collaboration with our colleagues at Unity Japan. The work done by the Marza & Unity Japan teams during this production created much of the foundation for Unity’s capabilities in Real time animation today, including Alembic support, the Unity Recorder (based off of the original Frame Capture system), as well as the first implementation (that I’m aware of) of a Fur shader in Unity, all of which we used as the foundation for the Sherman project.
Blocking out the Short
The conceptual blocking for the short was started by our animator, Bryan Larson, while most of the team was preparing for Unite LA last fall. You may be familiar with Bryan’s work from his career at Blue Sky, where he animated classic characters like Scrat from Ice Age among many others. Using the rough character sculpts created by Steven Shmuely, Bryan quickly created a first animation blocking pass to convey the basic story that he envisioned.
With the rough animation pass in hand, Mark Droste brought the animation into Unity and using Cinemachine – Unity’s procedural camera system – quickly blocked out a first camera pass and within a day or so had the first rough animatic to show the team.
Short clip from the Sherman Animatic, the artists were able to refine the story very quickly with rough assets
Unity’s powerful animation and camera tools allowed the team to go from rough idea to completed animatic in a period of days. Being able to rapidly iterate and try out ideas in real time is extremely powerful, and allows the creative team to really ‘find the story’ very quickly.
Once we had a completed animatic of the short, the team was able to truly get a sense of everything that would be needed to bring the short to life.
Lookdev with HDRP
Continuing from our work on Baymax Dreams, the decision to use HDRP for Sherman was an easy one. The team is very familiar with the renderer, and have the graphics engineering resources to dedicate to customization and extending the renderer as we needed.
We deliberately set a high visual quality bar for the short. We were very inspired by the recent Dreamworks short ‘Bilby’ as well as Unit Image’s latest Disney commercial, both having really well executed fur. We wanted to achieve that level of fur quality inside Unity, with a quality slider bar allowing for both a super fast working environment and the highest quality rendered output.
We used a global Post Processing Profile for the overall colour grading looks in conjunction with Cinemachine camera clip profiles for per-shot Post controls like exposure, DoF, vignetting – all the ‘lens like’ effects. This colour grading per scene/beat combined with per-shot lens PPS combination works super well as it’s possible to quickly try out different color grading tweaks across multiple shots while still preserving all your lens based looks.
Throughout production we would compare our current visuals next to our reference images using the Cinemachine Storyboard extension. We used the SplitView feature from Storyboard to view our visual target reference image over our project and used the Waveform Monitor to help us see how our frame differed colour-wise from the visual target. It’s powerful to be able to do this level of colour refinement per-shot in real time.
Our team only has a single 3D modeler (Steven Shmuely), so we knew that we would need to keep the environment fairly simple, without a massive number of assets. All of the characters & props in Sherman were modeled in Autodesk Maya (a few used 3ds Max as well), and were surfaced in Substance Painter. With the exception of a few assets that were borrowed from the Book of the Dead environment pack (modified to match the style & color palette by Steven later during the project).
The vast majority of the assets in Sherman use the standard HDRP Lit shader, an extremely versatile shader that provides a wide range of options for achieving the look that you are trying to achieve.
Starting with the characters, Steven quickly modeled a few versions of the Raccoon and other hero assets (Sprinkler, Dog bowl etc).
The video below shows some early WiP renders of these hero assets that were created during the production.
Another of our favorite assets is the garden Gnome. Materials for the Gnome were painted in Substance Painter, as shown in the image below.
During the lookdev stage, we would render quick turntables in Unity for review and signoff.
The Gnomes use the standard HD/Lit shader, and are set up like the image below:
Shader Graph for Materials
In addition to using the standard HD Lit material, a number of elements in the scene needed more advanced materials. Unity’s visual node-based material editor, Shader Graph, was used to craft specific materials.
For the hose pipe, Steven built a custom Shader Graph using the Stacklit shadergraph master node to create two independent specular lobes, one for the smooth reflections of the outer plastic coat of the pipe, and one for the inner layer with the fibre weave. The hosepipe shadergraph also includes independent controls to animate bubbles, simulated caustics and surface waves (that deform the surface using vertex animation). All of these exposed parameters can be independently keyed on the timeline, helping to bring a dynamic and layered look to the expanding hose pipe scene.
ShaderGraph was critical to allow Steven to be able to create and iterate on the hosepipe look without needing the help of a graphics engineer.
Whenever we are working on projects that involve characters, a critical element is eye shading and rendering. For Sherman, we experimented with a few variants of eye shaders before settling on the final result, which is a combination of a custom shader and a corresponding MonoBehaviour script to control the parallax direction for the eyes.
Eyes are much more complex than they seem initially. In many real time productions, they are often simplified to simple double shell geometry representing the inner and outer shell of the eye, but this results in much lower quality eyes. Since we had a number of close up shots of both Sherman (the bird) and the Raccoon in the short, we needed something more.
When attempting to tackle high quality eye shading, the most important thing to start with is the actual authoring of the eyes themselves. In order for the shader to be able to properly apply the parallax effect needed for the eye shading, we start with the Eye’s UV’s in the 0-1 range, like so:
The upper right is for the back-side of the eyeball, the main segment is for the front-facing eye, centered at 0.5,0.5 in UV space. Having the eyeball set up with UV like this allows the shader to modify the UV for the parallax effect, as shown below:
Another important detail is that bone orientation for the eyes is used in the calculation for the parallax effect a well. This is important to give us a frame of reference for the camera view direction to compare against. The refraction in the eye is a function of the two vectors (eye direction vs camera view direction) to indicate how much parallax effect to apply.
The final visual detail that John added for the eye shading is ambient occlusion for the eyes. This was done by raymarching the eyelids and exposing a parameter to allow Steven to control the amount of AO to apply.
The end result of the above was exposed in a MonoBehaviour ‘eye controller’ script that allowed the team to author and animate the eye properties per shot in the scene, giving full control over the eye iris details as needed to achieve the final result.
We’re very pleased with the final result, which is available with the full Sherman project. We are also looking to package up the shader & controllers as a stand-alone package for you to use in your own projects.
Camera Layout with Cinemachine
Mark Droste used Cinemachine to layout all of the cameras for Sherman. We also used the FBX exporter (also available in Package Manager) to export camera paths from Unity out to Maya for the animator to animate against.
Initially, the short was shot fairly loose, to give Bryan plenty of room to work to let his animation breathe. As the animation moved from blocking to final, this allowed Mark to refine the camera shots & movement as well.
With the characters already in Bryan’s basic staging positions, Mark was able to block in cameras directly in Unity with Cinemachine. Procedurally tracking the raccoon through most shots, and experimenting with new cameras using multiple Cinemachine camera tracks in a single Timeline allowed Mark to discover which shots were working well, and which could be pushed for more exciting compositions much quicker than animating the camera in traditional tools like Maya.
Using a combination of Cinemachine camera clips and animation tracks to animate camera properties, Mark was able to quickly lay down the cameras for the short and rapidly iterate to achieve the final results.
This wraps the first part of our blog series on Sherman. I hope that you enjoyed hearing about the production and the techniques and tricks that our team used to bring the project to life. Check out Part 2 for a deep dive into how we used Alembic for the character animations, and to learn more about the Fur implementation that we created for Sherman as well as some of the additional tools that the Innovation Group created to help team’s working on Linear Animation with Unity!
We are very excited about Sherman, and look forward to hearing more about how you are using Unity for Animation! Oh, and one final note: Sherman isn’t the Raccoon, he’s actually the cute fluffy Bird!
If you are interested in learning more about how Unity can be used for your animation projects, Unity’s EDU team can provide private on-site training workshops that can be fully customized for your or your team’s needs. Each workshop is led by a Unity Certified Instructor and features hands-on projects that teach Unity skills as well as best practices for implementation. You can also jump into our forums to discuss this blog post here.
For more information about Sherman, including access to the full project, go to our Film Solutions page, and get in touch to discuss how Unity and the Innovation Group can help bring your projects to life!