On September 17, 2017, the first version of Unity Machine Learning Agents Toolkit (ML-Agents) was released. The mission was simple – allow game developers and AI researchers alike to use Unity as a platform to train and embed intelligent agents using the latest advancements in machine learning. Since our initial “Hello World” release, we’ve seen the community and development of the toolkit grow significantly, with the project amassing over 8400 GitHub stars. Today, after more than two and a half years of development and 15+ release updates, we are excited to announce that the ML-Agents Unity package has reached v1.0 and is available as a Preview package. We’re also launching a new product and resources page for ML-Agents.
To give a bit of background – ML-Agents is an open-source project that enables games and simulations to serve as environments for training intelligent agents. It includes a C# SDK to set up a scene and define the agents within it, and a state-of-the-art machine learning library to train agents for 2D, 3D, and VR/AR environments.
Today, ML-Agents reached a significant milestone in becoming production-ready: the C# SDK has graduated to its first major version, v1.0, and is now available as a Preview package, com.unity.ml-agents, through the Unity Package Manager. This offers three key benefits to our users:
- API stability. The ML-Agents C# SDK has evolved over more than two years of development and testing with our own demo Unity environments (ranging from locomotion tasks to multiplayer games and robotic arms) to real games made with Unity. The result is a flexible, feature-rich, and stable API that is easy to integrate into any game or Unity environment.
- Ease of installation. You can now get started with the ML-Agents Unity package directly from the Package Manager without the need to clone the GitHub project.
- Verified Unity package (coming soon). The ML-Agents Unity package is now on track to become a Verified package for the 2020.2 version of the Unity Editor.
This blog post will walk you through a brief history of ML-Agents, overview our latest milestone release, ML-Agents Release 1, and offer a peek into upcoming ML-Agents improvements.
A brief history of ML-Agents
In our inaugural blog post, we wrote:
As the world’s most popular creation engine, Unity is at the crossroads between machine learning and gaming. It is critical to our mission to enable machine learning researchers with the most powerful training scenarios, and for us to give back to the gaming community by enabling them to utilize the latest machine learning technologies.
Over the following two and half years, ML-Agents stayed true to its mission and evolved in the nexus of the Gaming and Machine Learning communities. We continuously improved the toolkit by adding new training options such as Curriculum Learning, enabling the Curiosity module for sparse-reward environments, improving the speed and efficiency of training, enabling Self-Play for multi-agent scenarios, and providing native, cross-platform inference support for all models trained with ML-Agents. (See our GitHub ML-Agents Overview page for a description of all the ML-Agents features.) We’ve seen firsthand how these improvements have enabled new demos and environments such as Puppo and integrations with real games such as Jam City’s Snoopy Pop and Carry Castle’s Source of Madness.
Specifically for AI researchers, we created the Obstacle Tower Environment and Challenge (built with ML-Agents) to serve as a benchmark for vision, control, and planning. We’ve also seen other researchers and labs adopt ML-Agents for creating research benchmarks. Prominent examples include the Arena multi-agent environments, the Animal AI Olympics, and the continuous control Marathon Environments.
Since our initial release, the ML-Agents community grew from a loose confederation of AI researchers, game developers, and students to thousands of engaged community participants. This includes several creative demonstrations of ML-Agents that are shared online and directly with us. As a result, our GitHub repository has obtained over 8,400 stars and 2,300 forks, and our research paper Unity: A General Platform for Intelligent Agents was cited over 115 times.
ML-Agents Unity Package v1.0
Today, we’re delighted to announce that as of our latest release the ML-Agents Unity package is now at v1.0.0 and available as a Preview package in the Unity Package Manager. Our Unity: A General Platform for Intelligent Agents research paper has been updated to reflect ML-Agents Release 1.
This version change is reflective of two core improvements to ML-Agents:
- Package organization. ML-Agents is now four packages: one Unity package and three Python packages that include the Unity to Python communication and machine learning algorithms. Unless you are a researcher modifying our training algorithms or building your own, you will only interface with ML-Agents by integrating the Unity package into your project and using our training CLI.
- API improvements. Several months ago, we began an overhaul to make our C# SDK much easier and more intuitive to use. The result was a number of major improvements that were phased in over several releases, starting from v0.10.0 in September 2019 and culminating in our latest release. More details on these changes can be found in our release notes.
Starting with ML-Agents Release 1, each GitHub release will publish a new version for each of the packages that make up ML-Agents. With ML-Agents Release 1, we’ve set the following versions:
- com.unity.ml-agents (v1.0.0, C#)
- Public APIs have been defined and will follow semantic versioning for future releases. Additionally, the package has been added to the Unity Package Manager as a Preview package. We are planning to release a Verified package for Unity 2020.2.
- Communicator (v1.0.0, C#/Python)
- The communication protocol between Unity and Python has been stabilized and will follow semantic versioning. The functionality will be backward and forward compatible between both C# and Python packages that use the same major version of the communications protocol.
- ml-agents, ml-agents-envs, gym-unity (v0.16.0, Python).
- Python packages will continue to evolve and improve in a non-backward compatible way. While future improvements to the CLI may cause breaking changes, we plan to finalize and stabilize the CLI soon.
More details on the versioning for our packages and corresponding changes to our GitHub releases can be found on the Versioning page.
ML-Agents Release 1 is the beginning of a very exciting journey. We plan to continue developing the toolkit by improving the performance and efficacy of our training algorithms, evolving our Unity package based on your feedback, and adding more example environments that are inspired by gaming and industrial applications of ML-Agents. You can keep up and provide input on our roadmap in the ML-Agents Forum.
In addition to evolving the core toolkit, we have a number of exciting improvements planned that we’ll now briefly overview.
Verified package for Unity 2020.2
Part of being a verified package in Unity is to provide peace of mind that the package will have undergone significant testing and have been verified to work safely with a specific version of Unity. Meaning, if you want to use ML-Agents in a production game, we want to ensure we are supporting you. We are planning to release a Verified package for Unity 2020.2. This also means that we will be supporting the ML-Agents Unity package for the Unity 2020 LTS cycle.
One common piece of feedback we get from ML-Agents users is the desire for training without the need to install Python. We’ve also seen how restrictive it can be for our users to train on a local machine, limiting the number of environment and hyperparameter variations that can be trained at once. ML-Agents Cloud is a cloud offering we intend to launch later this year that will enable ML-Agents users to train on our scalable cloud infrastructure. Users will be able to submit many concurrent training sessions or easily scale out a training session across many machines for faster results.
Currently, we are opening up signups to be considered for early preview access to ML-Agents Cloud. If you are an existing ML-Agents user and need help with scaling and managing experiments, sign up here.
ML-Agents in a DOTS World
Unity’s core is being rebuilt with the Data-Oriented Technology Stack (DOTS). DOTS offers significant performance benefits by enabling builds to be smaller in size and run faster. These benefits are magnified in the context of simulations and machine learning. As such, we’ve been internally prototyping a version of ML-Agents that is built for DOTS. We’ve successfully integrated our DOTS prototype into sample scenes (see above) and Unity demos such as MegaCity and TinyRacing. The results have been outstanding. We were able to train agents in complex and large environments such as MegaCity in just a couple of hours on a standard laptop. We intend to release an experimental version of ML-Agents for DOTS later this year.
If your game or Unity project is being built using DOTS and you are interested in ML-Agents, please email us. We are looking for interested preview users to work with us on improving ML-Agents for DOTS.
For our robotics researchers, Unity with NVIDIA PhysX 4.0 has dramatically improved the quality of robotics simulation (see Physics Update in Unity 2019.3). Unity 2020.1 includes a new articulation joint system powered by Nvidia’s PhysX 4.1, which offers a dramatic improvement in simulating robotic arms and continuous joints. It uses Featherstone’s algorithm, reduced coordinate representation, and a new non-linear iterative solver to drastically reduce unwanted stretch in the joints. In practice, this means that we can now chain many joints in a row and still achieve stable and precise movement.
You can already get started with the new articulation joint system in Unity 2020.1 (beta). Check out the Unity Robotics Demo project (integrated with ML-Agents Release 1) if you’d like to experiment with the robotics environment above or use it as a sample project to create your own robotics environments. Additionally, we plan to expand the example environments within ML-Agents to include additional robotics and continuous control environments.
Learn how to implement ML-Agents Release 1
We’ve partnered with Immersive Limit to create ML-Agents: Hummingbirds, a course on the Unity Learn platform that teaches you how to implement ML-Agents Release 1 through exercises, code walkthroughs, and helpful discussions.
Learn how to train neural networks to perform a challenging task — getting hummingbirds with six degrees of freedom and complicated flight paths to their flowers. By the end of this course, you’ll have a good understanding of how you can harness the power of ML-Agents Release 1 to create intelligent agents and integrate them into your own Unity games and simulation projects.
On behalf of the entire Unity ML-Agents team, we want to thank you all for your continued support throughout the years and continuing on this journey with us!
To get started with ML-Agents, check out our GitHub homepage.
If you use any of the features provided in this release, we’d love to hear from you. For any feedback, general issues, or questions regarding ML-Agents, please reach out to us on the ML-Agents forums or feel free to email us directly. If you encounter any bugs, please reach out to us on the ML-Agents GitHub issues page.
If you’d like to work on this exciting intersection of Machine Learning and Games, we are hiring for several positions; please apply!