Search Unity

This post explains the technology behind Mecanim Humanoids. How it works, strengths and limitations, why some choices were made and hopefully some hints on how to get the best out of it. Please refer to Unity Documentation  for general setup and instructions.

HumanoidRig_pngHumanoid Rig and Muscle Space

Mecanim Humanoid Rig and Muscle Space are an alternative solution to the standard skeleton node hierarchies and geometric transforms to represent humanoid body and animations.

The Humanoid Rig is a description on top of a skeleton node hierarchy. It identifies a set of human bones and creates a Muscle Referential for each of those. A Muscle Referential is essentially a pre and post rotation with a range and a sign for each axis.

arm_stretch_in-pngA Muscle is a normalized value [-1,1] that moves a bone for one axis between range [min,max]. Note that the Muscle normalized value can go below or over [-1,1] to overshoot the range. The range is not a hard limit, instead it defines the normal motion span for a Muscle. A specific Humanoid Rig can augment or reduce the range of a Muscle Referential to augment or reduce its motion span.

The Muscle Space is the set of all Muscle normalized values for the Humanoid Rig. It is a Normalized Humanoid pose. A range of zero (min= max) for a bone axis means that there is no Muscle for it.

For example, the Elbow does not have a muscle for its Y axis, as it only stretches in and out (Z-Axis) and roll in and out (X-Axis). In the end, the Muscle Space is composed of at most 47 Muscle values that completely describe a Humanoid body pose.

arm_stretch_out_png

One beautiful thing about Muscle Space, is that it is completely abstracted from its original or any skeleton rig. It can be directly applied to any Humanoid Rig and it always create a believable pose.  Another beautiful thing is how well Muscle Space interpolates. Compare to standard skeleton pose, Muscle Space will always interpolate naturally between animation key frames, during state machine transition or when mixed in a blend tree.

Computation-wise it also performs as the Muscle Space can be treated as a vector of a scalar that you can linearly interpolate as opposed to quaternions or Euler angles.

An approximation of human body and human motion

Every new skeleton rig built for a humanoid character or any animation captured will be an approximation of the human body and human motion. No matter how many bones or how good your MOCAP hardware is, the result will be an approximation of the real thing.

Riggers, game companies, schools or software firms will propose their own version of what they thinks best represent the human body and motion and what will best fit their production needs.

The elaboration of Mecanim Humanoid Rig and Muscle Space was confronted to some hard choices. We had to find a compromise between fast runtime and animation quality or openness and standard definition.

2 Spine bones

spineThis is a tough one. Why 2, not 3? or an arbitrary number of spines bones?  Lets discard the latest, it is not about biomedical research. (Note that you can always use a Generic Rig if you absolutely need this level of precision). One spine bone is clearly under defined.

Adding a second one brings you most of the way. A third or even a forth one will only give a small contribution to the final human pose. Why is this? When looking at how a human spine bends, you will notice that the part of spine that is on the rib cage is almost rigid. What remains, is a main flexion point at the base of the spine and one other at the base of the rib cage. So there are two main flexion points. Looking at a contortionist even in extreme poses clearly show this. Considering all of this we decided to have 2 spine bones for the Humanoid Rig.

1 Neck Bone

This one is easier than for spine. Note that many game skeleton rigs don’t even have a neck bone and manage to do the job with only a head bone.

Rotation DoF

As with most skeleton rigs (it is even more often the case for games), the Mecanim Humanoid Rig only supports rotation animation. The bones are not allowed to change their local translation relative to their parent. Some 3D packages induce a certain amount of translation on bones to simulate elasticity of articulations or squash and stretch animation. We are currently looking at adding translation DoF as it is a relatively cheap way in term of computation performance to compensate for animation quality on less detailed skeleton rigs. It would also allow users to create retargetable squash and stretch animation.

0_twistpng

No twist bones

Twist bones are often added to skeleton rigs to prevent skin deformation problems on arms and legs when they are in extreme twist configuration.

Twist bones help to distribute the deformation induced by twist from start to end of the limb.

In the Muscle Space, the amount of twist is represented by a Muscle and it is always associated with the parent bone of a limb. Ex: The twist on the forearm happens at the elbow and not on the wrist.

50_twistpngHumanoid Rigs don’t support twist bones, but Mecanim solver let you specify a percentage of twist to be taken out of the parent and put onto the child of the limb.It is defaulted at 50% and greatly helps to prevent skin deformation problem.

Humanoid Root and Center of mass

Now, what would be the best way to represent the position and orientation of human body in world space?

The top most bone in hierarchy (usually hips, pelvis or whatever it is called) is where lies the world space position and orientation curves in a standard skeleton rig. While this works fine for a specific character, it becomes inappropriate when doing retargeting since from one skeleton rig to another the top most bone usually have a different position and rotation relative to the rest of the skeleton.

backflip_pngThe Muscle Space uses the humanoid center of mass to represent its position in world space. The center of mass is approximated using a human average body parts mass distribution. We do the assumption that, after scale adjustments, the center of mass for a humanoid pose is the same for any humanoid character. It is a big assumption, but it has shown to work very well for a wide set of animations and humanoid characters.

It is true that for standing up or walking animations, the centre of mass lies around hips, but for more dynamic motion like a back flip, you can see how body moves away from the centre of mass and how the centre of mass feels like the most stable point over the animation.

Body orientation

Similar to what the centre of mass does for Muscle Space world space position, we use an average body orientation for world space orientation. The average body orientation up vector is computed out of the hips and shoulders middle points. The front vector is then the cross product of the up vector and average left/right hips/shoulders vectors. It is also assumed that this average body orientation for a humanoid pose is the same for all humanoid rigs. As for the centre of mass, an average body orientation tends to be a stable referential as lower and upper body orientation naturally compensates when walking, running, etc.

Root Motion

A more in depth paper about root motion will follow, but as an introduction, the projection of the centre of mass and average body orientation is used to automatically create root motion. The fact that the centre of mass and average body orientation are stable properties of humanoid animation leads to a stable root motion that can be used for navigation or motion prediction.

The scale

One thing is still missing in Muscle Space to be a completely normalized humanoid pose… the overall size of it. Again we are looking for a way to describe the size of a humanoid that does not rely on a specific point like head bone position since it is not consistent from rig to rig. The center of mass height for a humanoid character in T-Stance is directly used as its scale. The center of mass position of the Muscle Space is divided by this scale to produce the final normalized humanoid pose. Said in another way, the Muscle Space is normalized for a humanoid that has a centre of mass height of 1 when in T-Stance. All the positions in the Muscle Space are said to be in normalized meters.

Original hands and feet position

When applying a Muscle Space to a Humanoid Rig, hands and feet may end up in different position and orientation from the original animation, due to the difference in proportions of Humanoid Rigs. This may result in feet sliding or hands not reaching properly.  This is why Muscle Space optionally contains the original position and orientation of hands and feet. The hands and feet position and orientation are normalized relative to Humanoid Root (center of mass, average body rotation and humanoid scale) in the Muscle Space. Those original positions and orientations can be used to fix the retargeted skeleton pose to match the original world space position using an IK pass.

IK Solver

ikdiff_pngThe main goal of IK Solver on arms and legs is to reach the original hands and feet position and orientation optionally found in the Muscle Space.  This is what happens under the hood for feet when “Foot IK” toggle is enabled in a Mecanim Controller State.

In these cases, the retargeted skeleton pose is never very far from the original IK goals. The IK error to fix is small since it is only induced by difference in proportion of humanoid rigs. The IK solver will only modify the retargeted skeleton pose slightly to produce the final pose that matches original positions and orientations.

Since the IK only modifies slightly the retargeted skeleton pose, it will rarely induce animation artefacts like knee or elbow popping. Even then, there is a Squash and Stretch solver, part of IK solver, that is there to prevent popping when arms or legs come close to maximum extension. By default the amount of squash and stretch allowed is limited to 5% of the total length of the arm or leg. An elbow or knee popping is more noticeable (and ugly) than a 5% or less stretch on arm or leg. Note that squash and stretch solve can be turned off by setting it to 0%.

A more in depth paper about IK rigs will follow. It will explain how to handle props, use multiple IK passes, interaction with environment or between humanoid characters, etc.

Optional Bones

The Humanoid Rig has some bones that are optional. This is the case for Chest, Neck, Left Shoulder, Right Shoulder, Left Toes and Right Toes. Many existing skeleton rigs don’t have some of the optional bones, but we still wanted to created valid humanoids with those.

The Humanoid Rig also supports LeftEye and RightEye optional bones.  Eye bones have two Muscles each, one that goes up and down and one to move in and out. The Eye bones also work with Humanoid Rig LookAt solver that can distribute look at adjustments on Spine, Chest, Neck, Head and Eyes. There will be more about LookAt solver in the upcoming Humanoid IK rig paper.

Fingers

Finally, the Humanoid Rig supports fingers. Each finger may have 0 to 3 digits. 0 digit simply means that this finger is not defined. The are two Muscles (Stretch and Spread) for the first digit  and one Muscle (Stretch) for 2nd and last digit. Note that there is no solver overhead for fingers when no fingers are defined for a hand.

Skeleton rig requirements

In-between bones

In many case, skeleton rigs will have more bones than the ones defined by the Humanoid Rig. In-between bones are bones that are between humanoid defined bones. For example, a 3rd spine bone in a 3DSMAX Biped will be treated as an in-between bone. Those are supported by Humanoid Rig, but keep in mind that in-between bones won’t get animated. They will stay at their default position and orientation relative to their parent defined in the Humanoid Rig.

Standard Hierarchy

The skeleton rig must respect a standard hierarchy to be compatible with our Humanoid Rig. The skeleton may have any number of in-between bones between humanoid bones, but it must respect the following pattern:

Hips – Upper Leg – Lower Leg – Foot – Toes

Hips – Spine – Chest – Neck – Head

Chest – Shoulder – Arm – Forearm – Hand

Hand – Proximal – Intermediate – Distal

The T-Stance

tstance_pngThe T-Stance is the most important step of Humanoid Rig creation since muscles setup is based on it. The T-Stance pose was chosen as reference pose since it is easy conceptualize and that there is not that much room for interpretation of what it should be:

– Standing straight facing z axis

– Head and eyes facing z axis

– Feet on the ground parallel to z axis

– Arms open parallel to the ground along x axis

– Hands flat, palm down parallel to the ground along x axis

– Fingers straight parallel to the ground along x axis

-Thumbs straight parallel to the ground half way (45 degrees) between x and z axis

When saying “straight”, it does not mean bones necessarily need to be perfectly aligned. It depends on how skin attaches to skeleton. Some rig may have the skin that looks straight, but underneath skeleton is not. So it is important that the T-Stance be set for final skinned character. In the case you are creating a Humanoid Rig to retarget MOCAP data, it is good practice to capture at least of few frames of a T-Stance done by the actor in the MOCAP suite.

Muscle range adjustments

By default muscle ranges are set to values that best represent human muscle ranges. Most of the time, they should not be modified. For some more cartoony character you may want to reduce the range to prevent arms entering body or augment it to exaggerate legs motion. If you are creating a Humanoid Rig to retarget MOCAP data you should not modify the ranges since the produced animation clip will not respect default.

Retargeting and Animation Clip

Mecanim retargeting is split into two phases. The first phase consists of converting a standard skeleton transforms animation to a normalized humanoid animation clip (or Muscle Clip). This phase happens in the editor when the animation file is imported. It is internally called “RetargetFrom”. The second phase happens in play mode when Muscle Clip is evaluated and applied to the skeleton bones of a Humanoid Rig.

muscleclip_pngIt is internally called “RetargetTo”.There are two big advantages of splitting retargeting into two phases. The first one is solving speed. Half of the retargeting process is done offline, only the other half is done at runtime. The other advantage is scene complexity and memory usage. Since the Muscle Clip is completely abstracted for its original skeleton, the source skeleton does not need to be included in runtime to perform the retargeting.

The second phase is straight forward. Once you have a valid Humanoid Rig, you simply apply Muscle Clip to it with RetargetTo solver. This is done automatically under the hood.

The first phase, converting a skeleton animation to a Muscle Clip, may be a bit trickier. The skeleton animation clip is sampled at a fixed rate. For each sample, the skeleton pose is converted to a muscle space pose and a key is added to the Muscle Clip. Not all the skeleton rig will fit, there are so many different ways a skeleton rig can be built and animated. Some skeleton rig will produce a valid output, but with possible loss of information. We will now review what is needed to create a lossless normalized humanoid animation… the Muscle Clip.

Note: By lossless we mean that retargeting from a skeleton rig to Muscle Clip and then retargeting back to the same skeleton rig will preserve the animation intact. In fact, it will be almost intact. The original twist on arms and legs will be lost and replaced by what the Twist solver computes. As explained earlier in this document, there is no representation of twist repartition in Muscle Space.

  • The local position of bones must be the same in the humanoid rig and in the animation file. It happens that the skeleton used to create the Humanoid Rig differs from the one in the animation file. Be sure to use exactly the same skeleton. Warnings will be sent to the console at import, if it is not the case.
  • In-between bones must have no animation. This often happen for a 3DSMAX skeleton where the 3rd spine bone has both translation and rotation animation on it. It also happens when Bip001 is used as Hips and that Pelvis has some animation on it. Warnings will be sent to console at import if it is not the case.
  • The local orientation of the in-between bone must be the same in the Humanoid Rig and in the animation file. This may happen when using Humanoid Auto Configure that relies on Skin Bind Pose to create T-Stance. Make sure that Skin Bind Pose rotation for in-between bones is the same that one found in the animation file. Warnings will be sent to console at import if it is not the case.
  • Except for Hips, translation animation is not supported on bones. 3DSMAX Biped sometimes puts translation animation on spine bones. Warnings will be sent to console at imp or if it is not the case.

The 3DSMAX Biped is pointed as a problematic rig here. It is probably because of its popularity and the fact that we had to support many cases of it being used with Mecanim. Note that if you are going to create new animations to be used with Mecanim Humanoid Rig, you should follow the rules stated above from the start. If you want to use already existing animation that break some of the rules, it is still possible, the Mecanim retarget solver is robust and will produce valid output, but the lossless conversion can’t be guarantied.

Note that if you are going to create new animations to be used with the Mecanim Humanoid Rig, you should follow the rules stated above from the start. If you want to use already existing animation that breaks some of the rules, it is still possible as the Mecanim retarget solver is robust and will produce valid output, but the lossless conversion can’t be guaranteed.

29 Comments

Subscribe to comments

Comments are closed.

  1. Having the option to add a 3rd spine would be amazing.

    In our case, we have about 1000 combat animations that were built using 3 bones in the spine and while some of them are ok, there’s a lot of twisting and stretching in them that are causing the animations to look wildly incorrect. We’ve tried generic as an alternative and that fixes the animations making them look great but it brings with it new physics/collision problems.

  2. Bob,

    Thanks for the commented reply. We typically put the ribcage bone at the sternum and 2 additional spine bones below that. Unfortunately the clavicles couldn’t help with getting the shapes through the spine that 3 bones allows for.

    Michael,

    we’ve solved the forearm twist issue with creating a component script that applies a defined %(default 50%) of the axial rotation from the wrist to the roll bone and we’ve turned off the twist in the muscles group that was trying to compensate at the elbow

  3. Pure Muscle Pro And Muscle Factor X

    June 18, 2014 at 11:43 pm

    Hi there, I read your blogs like every week. Your writing style is
    witty, keep up the good work!

    My web page; Pure Muscle Pro And Muscle Factor X

  4. I wish I could edit my post… I already went back and fixed my clavicle translation problem. Wasn’t as big a deal as it should’ve been (but Maya might be playing ticks on me). Bigger problems for me now are dealing with the uncertainty of the twist solver, as I have an extra joint for forearm rotation, and I think I have an extra spine segment that you don’t (my hierarchy is: root – spine1-spine2-ribcage-neck-head ; and root-hips-thigh-knee-foot-toe ). The hierarchy you list starts with hips, no mention of root joint. That could be another problem. The root joint could be ignored, but then we’d lose root motion, right? Were we supposed to make hips the root, because no rigging scripts in Maya I’ve used do this.

    This wouldn’t be a big deal if this info had come out when Mecanim was released, but even now the unity docs on importing models and animation don’t have any of this very important info (no root in hierarchy; 1 spine only; rotate forearm from elbow/no twist bone; no translation). Wish I’d known that 6 months ago when I was rigging and skinning and didn’t have a small library of animations to risk compromising.

  5. Not sure my rig would help you, there’s nothing special about it just basic maya joints, no IK, constraints or anything special. I’m personally not using any squash and stretch at the moment. For some extreme poses I had to translate the shoulders (clavicle), toes and neck to get it to look right (the shoulders is the rig’s fault- the clavicle joints are too far from the spine- but fixing that would undo months of skinning and animation work). I should probably just go and delete all non-root translation keys and then see what I can fix with rotation, but it really won’t look too good.

    My previous comments were more geared towards the future, I didn’t expect immediate action (so kudos on that!).

    – “Applying the the Muscle Clip to the original skeleton would preserve the exact same translations.”
    This is a little confusing to me, perhaps because I don’t totally understand muscles yet. Work flow wise, are we talking about an additional animation I’d have to make in unity to fix the lack of translation? Or are we talking about a potential work around, so that on import the original translation data gets preserved as muscle data somehow (ie, the process is automated)?

    Also, as this is turning into more of a conversation, should we maybe move this to a forum post?

  6. Micheal,

    I started to work on “retargetable” translations or translation dofs. Lets say muscle translation. Translation will be normalized relative to muscle space.

    Applying the the Muscle Clip to the original skeleton would preserve the exact same translations. It would also retarget to other Humanoid rigs. This way, we could retarget squash and stretch animation.

    Some users already provided me with sample content to test this. I could test with your rig/animation if you want to share?

  7. Bob,
    First thanks for taking the time to reply.
    Is there anyway we can preserve joint translation animation for the ” human specified bones” too? I’ve been using it to compensate for rigging problems in the shoulders (and maybe the neck too, I can’t remember). If imported sans translation, half of my arm animations will be broken.

    I know for many cases it might not be used, but there are a lot of circumstances where some squash and stretch is really necessary for good animation. Maybe put a toggle button somewhere in the import process to preserve translation on a given clip?

  8. With this asset, Animation Baker ( https://www.assetstore.unity3d.com/en/#!/content/18217 ) you can now use retargeted animations with Legacy.

  9. Bob,

    Thank you very much. :)

  10. TMPXYZ

    With a little bit of c# you should be able to convert a muscle clip back to skeleton animation.
    – Setup a humanoid rig for the “target” skeleton
    – Create a Controller with the muscle clip to convert
    – for each frame, animator.play(frame time); animator.update();
    – read the skeleton transfroms
    – write transform trs key to AnimationClip

  11. Trent

    2 Spine bones was a choice, mostly happy with it, but not completely. Could clavicle be use to compensate for upper rib cage?

    Where would preferably be this 3rd spine bone? middle of rib cage or on top of it?

  12. Michael,

    It only apply to human specified bones and bones that are in-between humanoid specified bones. All other bones may have translation animation

  13. “The bones are not allowed to change their local translation relative to their parent.”

    WHAT?! WHY?! I hope I am misunderstanding something. If not please fix this first!
    Does this only refer to avatar specified bones or all additional undeclared bones like a face rig?

  14. Thanks for explanation on a lot of this. I’d still have to disagree with your choice to use 2 rather than 3 spine bones however. When you are rotating along the same axis through the spine bones, very little difference is seen, like in your contortionist example. But, when you counter rotate on the same axis from bone to bone so much of the pose can be lost and the bits of weight in the ribcage in any contrapposto pose or even in the ribcage hitting during walk and run cycles doesn’t come through as well.

    Support for a 3 bone spine would be much better dealing with more detailed games.

  15. this is late post, we need more mecanim details.

  16. Great article, very helpful in thinking about the best way to retarget human characters.

    To reply to the comment above about using Blender. I wrote a blog post recently where I use Mixamo as a middle man to get better rigging:
    http://blog.diabolicalgame.co.uk/2014/05/blender-to-mixamo-to-blender-to-unity.html

  17. cap quang fpt

    May 28, 2014 at 2:06 pm

    We’ve have just playing with Mecanim beforr playing with Legacy The extra under-the-hood detail is really welcome us. Thank for best regard

  18. Hi, Bob,

    Thanks for your reply.

    I have a plugin that is used to create/modify rigged character animation in unity editor with Animation view, it works for Legacy and Generics rigs well, but for humanoid, that’d be not so easy.

    I had a hacky way to create animation on humanoid rig, but I cannot figure out how to modify an muscle clip without fbx file.

    So I doubt if there’s a way to convert the muscle clip “back” to generics or legacy animation, onto a dummy skeleton or something?

    I Would be grateful if you could shed more light on this one.

  19. Anomalous Underdog

    May 27, 2014 at 11:56 pm

    I make my own rigs from Blender and I’d appreciate tips on how to make sure my rig will survive after being imported into Mecanim.

    So twist bone isn’t supported, what then? I prefer creating animations in Blender and I need a twist bone so they won’t get that horrible deformation.

    It seems to me Mecanim needs one continuous hierarchy of bones for a limb, so I thought that was why twist bones didn’t work: they aren’t part of the bone chain.

    Also to anyone here using Blender for rigging/animating and using Mecanim, what’s your best practices?

  20. ELECMAN

    “Non-humanoid” rig is kind of vague and defeats the goal of having a standard format. This said, different things could be done:

    1 – An Inverse Knee toggle when you create the humanoid rig
    2 – Fully editable Muscle Referential when you create the rig, you could flip pre and post rotation or sign of upper leg, lower leg and foot
    3 – You create own named muscle referential on top of a random skeleton

    The last one may sound cool, but with it you would loose the benefit of having a standard humanoid format. The first one does not give lot of room for customization, but it keeps some kind of standard. 2nd one is an hybrid.

    So far we decided to stick with the humanoid standard. Note that you could always use Generic rig when you character does not match humanoid rig.

  21. TMPXYZ,

    You cannot modify the humanoid animation probably because it is an imported animation and that it is read only. Duplicate it (ctrl-d) and then you will be able to modify it. Note that there is no fancy keyframing rig for humanoid yet, but you can still edit numeric values in AnimationWindow.

  22. Victor, Partel

    We are looking for the best way to give access to “Retarget From”. Probably a Get and Set Muscle Pose. Mocap hardware vendor are also looking for this to enable live retargeting from mocap device. The Get and Set could also be use to modify muscle pose after State Machine eval and before RetargetingTo to procedurally modify a muscle pose. (I need to write something about humanoid evaluation pipe).

  23. Can we expect support for non-humanoid rigs at some point? I have a Mech with backward bent knees. It would be awesome if I could re-target human motion capture data to that Mech.

  24. Is there a way to convert a muscle clip back to a standard transform animation?
    If available, what skeleton will this animation be applied on?

    I cannot modify the humanoid animation in the Unity Animation view, have been stuck on that for a long time…

    Could you give some hints on this one?

  25. Thank you for this! Its great to get some confirmation for what I thought was going on under the hood, it really helps to understand.

    I’d like to second to Victor’s request earlier, we need access to “Retarget From”, we need a way to convert .anim to a muscle clip, so we could create something where we could just drop in an AnimationClip and optionally an existing Avatar Definition and get a “Mecanimatable” clip in return.

    I’m creating a tool for animating characters entirely in Unity or procedurally modifying any existing animation, I want to make it a beautiful thing, but without this it will require terrible hacks to make it work with Mecanim…

    Thank you,
    Pärtel

  26. Wow, thank you for so detailed explanations. The great thing about Unity 3D development as a whole, and Mecanim Humanoids we actively use recently, is the great space for customization and great documentation, that makes it easy for developers to use and to customize all technologies to their needs

  27. Thanks for this! We’ve recently just started playing with Mecanim after playing with Legacy for so many years. The extra under-the-hood detail is really welcome!

  28. Wow, it’s really awesome that you share all those details including explanations why certain decisions were made. I think it’s really that kind of “deeper understanding” that makes working with Unity the greatest joy!!!

  29. Victor Lajide

    May 26, 2014 at 6:19 pm

    I was wondering If there was a way to write a custom model importer for a humanoid model that auto sets mecanim.
    Or is there a way to convert animations made in unity from generic to mecanim?