Visualizing BMW’s self-driving future
BMW employs Unity across its automotive lifecycle for a variety of use cases, from transforming production processes with AR and VR to marketing its vehicles in groundbreaking ways. Let’s explore one of BMW’s most innovative applications of real-time 3D technology – making it easier to navigate the complexity of autonomous driving (AD) and challenge its AD function across millions of simulated scenarios.
The BMW Group – home to the BMW, MINI, Rolls-Royce, and BMW Motorrad brands – has been working on highly automated driving (AD) since 2006. In the upcoming years, the company hopes to offer drivers a groundbreaking opportunity – to buy a vehicle they will almost never need to drive themselves.
The BMW Group is targeting to sell cars with Level 3-enabled automation for driver assistance systems, highway driving, and parking in the upcoming years. (SAE Level 3 is defined as conditional driving automation with some human intervention required.)
Driving 95% of all test miles virtually
Just 5% of all BMW’s test miles will be driven by actual vehicles (video credit: BMW).
Around the world, a fleet of test vehicles from the BMW Group will pressure-test this technology. Because this fleet cannot gather all of the data needed for AD development, nearly 95% of all BMW’s test miles are driven by virtual vehicles in virtual worlds.
These simulations take place at BMW’s Autonomous Driving Campus in Unterschleissheim, Germany, just north of Munich. Nicholas Dunning, a graphical simulation developer at the BMW Group, is part of the core 12-person development team that has built custom tools made with Unity to help the 1,800 AD developers at BMW’s campus visualize and advance their work.
“At BMW, we believe simulation is key for developing autonomous driving,” says Dunning. “Unity plays a pivotal role in helping our team create, visualize, and evaluate the millions of virtual road trips needed to help us achieve our AD ambitions.”
How BMW uses Unity for AD development
With the overwhelming majority of its testing taking place in BMW’s bespoke datacenter for AD development, BMW needed to give its AD developers an easy way to:
- Visualize the raw data from simulations in an immediately understandable, true-to-life way, beyond graphs and charts
- Evaluate the current state of their AD functions across countless simulated scenarios.
Taking advantage of Unity’s extensibility, Dunning’s team developed a custom Unity-based solution to address these needs. Let’s dive into the unique way they are using Unity to help the BMW Group bring a safe, reliable AD system to the street on schedule.
Making the process for creating scenarios quick and easy
BMW used Unity to develop a graphical scenario editor that vastly simplifies the process to test and validate features in development. The interface makes it easy for AD developers to visualize and set up thousands of simulated scenarios that increase feature maturity and readiness.
Here’s a sampling of various elements they can parameterize in the scenario editor to battle-test features in simulation:
- Quantity and type of traffic vehicles (car, bus, etc.)
- Traffic signs (ground or mounted)
- Lanes (straight, curved, etc.)
- Lane boundaries (none, single-solid, double-solid, dashed, etc.)
- Environmental conditions (time of day, fog density, precipitation level)
- Vehicle trajectory planning
In addition to scenarios generated manually by BMW’s developers, scenarios are also extracted from traffic scenes recorded by the test fleet. This data is post-processed and automatically converted into simulation scenarios. A further analytic step identifies scenarios that would be interesting to develop and vary them.
The video below shows a real-world scenario of a vehicle cut-in on a highway in Germany, as well as the converted scenario in the simulation. Because this was identified as an interesting scenario, it undergoes variations. In this case, these variations test the vehicle’s ability to maintain a safe distance from the cut-in car in various weather conditions, including rain, low sun position, and fog.
A simulated scenario converted from fleet testing is varied across weather conditions.
Visualize simulated tests in real-time
Using Unity as a visualization front end for simulated testing is highly beneficial to BMW’s AD developers. With real-time 3D, they have full control over how they interact with this immersive digital reality.
As shown in the video below, they can experience a real-time, connected shift in point of view as they alter their perspective of the vehicle or any other object within the virtual scene. They can zoom in for a closer inspection or move back to get a sense of scale, making it easy to get a holistic understanding of everything happening in the simulated scenario.
Unity lets BMW’s AD developers explore the simulated scenario from any vantage point. This scenario shows a vehicle surrounded by unknown objects (visualized as purple blocks) to help evaluate the AD function’s ability to operate with a mixture of known and unknown data.
Initially, BMW built highly detailed, realistic environments, but over time found that switching to a more abstract visualization style and only rendering key components (e.g., road, vehicles) helped to eliminate data noise and allowed AD developers to better concentrate on the results of each simulation.
Get contextual, instant feedback from testing
BMW’s AD developers can not only quickly create scenarios for testing, but get immediate, visual feedback on the readiness of their AD function. They can literally see how the vehicle performed during the test in real-time 3D, rather than having to parse through data in 2D charts and graphs.
The visualization and evaluation data (lower-right corner) are displayed and synced in real-time, making it easy for developers to analyze results in context.
As BMW continues its progress in its AD ambitions, Dunning and his team hope to eventually extend their Unity-based solution beyond its core audience of AD developers. The team sees tremendous potential in collaborating with their colleagues responsible for in-car testing to ensure the pre-production Level 3 vehicles perform as promised before they go into full production.
Stay tuned for Part Two, where we’ll share how BMW is using Unity to overcome the daily challenges of AD development.
Learn more about how Unity is used for AD simulation in our whitepaper: Top 5 Ways Real-Time 3D Is Revolutionizing the Automotive Product Lifecycle.