The road to 240 million virtual kilometers: BMW’s autonomous driving journey with Unity

Before BMW’s autonomous driving (AD) technology is mass production-ready, it will need to drive 240 million virtual kilometers. Learn how Unity is helping BMW put more on its odometer every day. In our first post, we covered how a team at BMW Fully Autonomous Driving & Driver Assistance Systems has used Unity to develop custom tools for simulation visualization and scenario creation. With these tools, the BMW Group is well-equipped to tackle the most daunting daily challenges in AD development.  Let’s walk through a few areas where Unity is helping out in a major way.

By combining simulation with key performance indicators (e.g., continually maintaining a safe distance from traffic vehicles), BMW can assess how complete its features actually are. As individual components of its AD function master basic scenarios, BMW can conduct mass validation of its entire AD system in more complex situations.

These tests come in multiple forms:

• Small-scale feature tests – These tests are the most common type of testing, and they enable BMW to rapidly evaluate portions of its AD system, such as vehicle trajectory planning. In a typical day, BMW’s team will log tens of thousands of virtual kilometers; the majority of these are short-distance tests (from hundreds of meters to 1 kilometer) in increments of less than a minute.
• Large-scale system tests – Instead of a series of mini-tests for a specific feature, this type of simulation is designed to test the broader AD system. It plays out as an extended scenario that can run for hours and strives to replicate real-world scenarios; for example, an autobahn trip between the German cities Munich and Stuttgart. These simulations are more complex, often involving a virtual world complete with moving vehicles, pedestrians, and variable weather conditions, as well as map data, sensor models as inputs to perception algorithms, vehicle trajectory planning, vehicle dynamics, and much more.

Because driving situations can be repeated as often as required in simulation, BMW runs tests throughout each day, including “night drives.” After using the Unity-based scenario editor to set up tests, developers can review the results the next morning, leverage analytics to pinpoint exactly when functions failed, and bring up the exact point of failure in a frame rendered from Unity. The team can automatically extract those problematic situations as small-scale scenarios, and then visualize them to better understand why the situation was problematic.

For instance, in this scenario, a traffic vehicle cuts in suddenly, yet the host vehicle does not decelerate immediately, resulting in a near accident. The scenario can be subsequently replayed after each incremental code update until the AD function reacts properly.

After an initial failure in this scenario, improvements ensure that this vehicle brakes properly in response to a traffic vehicle merging into its lane aggressively.

To reach the high automation level for their vehicles, BMW’s developers need to identify the limitations of their AD functions in as many situations as possible. Yet scenarios like the ones simulated in the video below are too cost-prohibitive, difficult, or dangerous to replicate in the real world.

Using the Unity-based scenario editor, the developers can devise edge-case scenarios, such as a traffic vehicle running a stop sign. Uncovering these corner cases in the confines of a virtual world is much more cost-effective and safe while allowing for the opportunity to test reproducibly.

BMW uses simulation to test scenarios that are too unusual to occur or too risky to create in a real-world driving environment. Here three edge cases are shown: 1) A pedestrian unexpectedly appearing in the host vehicle’s lane in a high-speed, highway setting; 2) A traffic vehicle cutting in suddenly; 3) A stopped vehicle in the host vehicle’s lane.

Unity is used on a daily basis to help the 1,800 AD developers in the BMW Group continuously improve the code for which they are individually responsible. As they make changes to the codebase to add a new function or improve an existing one, they run integration tests before merging into the master.

For instance, a developer focused on perception can use the Unity-based scenario editor to design multiple scenarios in which the vehicle passes a speed limit sign. These small-scale tests are simulated on the developer’s PC and can be visualized with Unity live as they are being run.

The developer can visually validate their results as well as use evaluation metrics to identify improvements or confirm that the feature is ready to be merged into the master (i.e., the vehicle adjusts to the posted limit every time).

Developers can simultaneously test and visualize the results of their incremental code updates. Post-merge, they can run acceptance tests to identify failures in other functions that arise as a result of their commit or vice versa. For instance, a merge from their peer could introduce a bug that affects the perception of speed limit signage. The developers can use Unity for visual debugging and easily inspect what is happening so they can fix things faster. BMW’s system is set up in such a way that developers can set breakpoints within the driving function and within the simulation code. The AD function and simulation are always in sync with one another, so the team can step through the code line by line and swap between the two worlds as they debug. The synchronicity is also mirrored by the visualization, allowing simultaneous inspection of the code and the simulated world. Because developers can still move around and inspect values in the Unity-based application, they can reduce the number of tools needing to be open at the same time, while still keeping the data as transparent as possible. All these elements ensure that the production code that will ultimately power BMW’s autonomous vehicles meets its standards for safety and reliability.

---

Check out Unity Industrial Collection or learn more about the ways that Unity is used for AD simulation in our whitepaper: Top 5 Ways Real-Time 3D Is Revolutionizing the Automotive Product Lifecycle.