All Unity Muse capabilities are now available in the Editor, plus 3 new updates

We’re excited to announce that Unity Muse users can now use all five Muse capabilities directly in the Unity Editor. By bringing AI abilities to where you create, you can now more easily and conveniently use Muse to complement your creative process and streamline development. 

While you may already be familiar with Muse Sprite for creating original 2D assets and the recently improved Muse Texture generation for creating a breadth of unique textures for 3D objects, this update brings three exciting improvements: 

• A new Editor-integrated version of Muse Chat provides project-aware responses.
• Now available to all users, Muse Animate allows you to create animations within the context of your project.
• LLM-powered Muse Behavior, which you can use to set up character interactions with an intuitive AI-assisted interface, is now available to anyone using Muse.

Following Muse’s launch last year, we asked for feedback on how we can continue to improve the chat experience. What we heard was an overwhelming desire to be able to troubleshoot errors without leaving the Editor. We received feedback that it would be helpful for Muse Chat to automatically know more details of a project to reduce the time spent giving context and get more relevant answers faster.  We’ve heard your feedback, and are excited to announce that Muse Chat is now available as an in-Editor package. This update also provides a foundation for providing project-aware responses. Muse Chat will now be able to retrieve key details about your project, including the Unity version, active render pipeline, input system used, target platforms, API compatibility, and other project settings.

One example of how this update will simplify troubleshooting issues is solving console errors. When you have an error, you can now click on it and ask “How can I fix this error?” Muse will automatically retrieve information on what’s causing the issue and provide a suggestion for how to resolve it. 

Muse Chat will also continue to provide usable scripts that you can plug and play, just as it has before. Over time, Muse Chat will continue to evolve and become a more intelligent chat-based assistant that can provide tailored answers and information, code snippets, and smarter integrations with the Editor. 

Animation is a complex craft. Prototyping basic movements capable of plugging into game mechanics early can save you a lot of time in later stages of development. Muse Animate is perfect for helping you rapidly generate humanoid animations through natural language. 

Similar to Muse Chat’s feedback, we commonly heard that Muse Animate needed to be more deeply integrated in-editor. Based on that resounding feedback, we’ve launched a new in-Editor Muse Animate prerelease package that’s available for all Muse users. With this new version, you can create Unity Animation Clips directly within the Unity Editor and easily retarget them to work with characters that use the Unity Humanoid Rig.

You can also edit your generated animations with Muse by selecting an animation to decompose it into multiple poses. From there, you can select effectors (little orange dots on some of the character’s joints) to more finely adjust the movements. Muse Animate will process your changes and regenerate the animation. The package also makes additional editing features available, such as extrapolated posing, loop to first pose, and transition duration.

With this release, we are also actively working on enhancements to the existing AI animation model and introducing new ways to generate and modify animations, including the ability to create animations from sketches or videos.

We’ve received a lot of requests for a built-in behavior tree that would allow you to bring game characters to life with NPC interaction abilities. Last year, we started a closed prerelease of Muse’s Behavior capability – our take on LLM-powered decision trees – to allow you to create custom logic for characters and objects in an easy-to-use workflow. This set of features also included generative abilities to help you set up more complex interactions.

Now, the Muse Behavior prerelease package with LLM features is available to all Muse users. We’ve designed and improved this package to make it easier to use and for more intuitive editing. Like a classic behavior tree model, Muse Behavior consists of nodes and branches that are human-readable, allowing you to create representations that read like stories.

Words within a node’s story are tagged as inputs used by the node’s internal logic. The words are then automatically converted into fields so you can embed data directly into the node. Want your character to speak? Add the talk node, provide the words and just like that, your character will have dialog that’s triggered by player interactions. 

Muse Behavior also includes a blackboard, which contains variables that can be reused in nodes across the graph. Simply click on the link button of a field, then choose from a menu of assignable options. Variables in the blackboard can be assigned values in code or assigned through the Behavior Graph Agent component in the Inspector window. This ability to set up and then duplicate repeatable actions makes it faster and easier to create complex and repeating interactions. 

With Muse Behavior, you can create custom actions that give you control over the high-level structure and narrative of your behavior trees. And don’t worry if you’re totally new to creating behavior trees – the node wizard guides you step by step through the process of creating a new action type and adding to the graph. You can also use the LLM feature to automatically generate your tree.As we continue to improve Muse Behavior, you can expect more generative features to further simplify complex behavior tree setups.

These new packages and improvements are just the start. We’re continuing to innovate on existing capabilities like Muse Texture, where we’re leveraging new original research and proprietary models to generate true PBR materials for 3D objects. This means that you’ll be able to produce a multi-material UV texture that can wrap onto an object directly, and it will react more accurately to lighting.

We also understand the importance of audio for setting the scene, so later this year, we’ll release Muse Sound so you can produce AI-generated, prompt-to-sound effects such as movement and environmental sounds.

At GDC, we showcased how you could use all five Muse capabilities together to customize a game loop in the garden scene of our URP sample project. Check out our session Unity Muse: Accelerating prototyping in the Unity Editor with AI to learn how you can use all of Muse’s abilities to quickly customize a project scene and gameplay.

We’ve updated the Muse onboarding experience to make it easier to start a free trial of Muse and add the Muse packages to your projects. Visit the new Muse Explore page to get started, and let us know what you think of the newest capabilities and improvements in Discussions.