Showcasing the world’s first photorealistic mixed reality demo by Varjo and Volvo
Get a behind-the-scenes look at an ambitious automotive project made with Unity from Varjo, the maker of industrial VR/XR headsets known for their superior visual quality in VR.
The team at Varjo is behind some of the most innovative projects in the world of mixed reality. They previously shared a photogrammetry-based environment they created in VR with us, and today they will share how they, together with Unity and Volvo, broke new ground on a demo that brings the real and virtual worlds together like never before.
Get a first hand look at this project in-person at Unite Copenhagen. Volvo and Varjo will be on-site showcasing this experience, as well as speaking in multiple sessions:
- How Volvo embraced real-time 3D and shook up the auto industry
- Creating next-gen VR and MR experiences using Varjo VR-1 and XR-1
- Future mobility, smart cars, and autonomous driving: Preparing for a new era at Volvo
The content below is courtesy of Varjo.
Mixed reality means blending virtual content with the real world. So far mixed reality has been accomplished with optical see-through, where the user sees digital objects augmented on top of reality through a pair of glasses. This is fine for portraying infographics or playing games, but for realistic scenes, it offers little value. Optical see-through devices can’t display black or opaque content on top of the real world. Everything appears hazy and holographic.
We at Varjo wanted to get rid of this limitation and be able to render photorealistic, opaque content – where it is impossible to distinguish between what is real and what is virtual. Our mission was to make photorealistic mixed reality possible with video pass-through. Video pass-through means using cameras to digitize the world in real-time, and then showing the combined result of real mixed with virtual to the user.
Before we could achieve this, we first needed a VR headset capable of displaying the real world in human-eye resolution. That is why we released our first human-eye resolution product VR-1, targeted for professional users, to the market in February 2019.
And at Augmented World Expo 2019 in Santa Clara, we showed a glimpse of the magic that can be accomplished with video pass-through. We publicly demonstrated our new headset XR-1 Developer Edition for the first time with a joint demo with Volvo, made with Unity. With XR-1, you can blend virtual content seamlessly with reality with extremely low latency and integrated eye-tracking, in superior resolution.
Here’s how the world’s first photorealistic mixed reality demo was created.
This video is unmodified material shot through the Varjo XR-1 Developer Edition. With XR-1, you can see photorealistic virtual content blending with reality in a full field of view. You can also switch seamlessly from XR to full VR.
Developing the XR-1 demo with Volvo and Unity
Varjo began working on a video-pass through mixed reality headset in early 2018. The collaboration between Varjo and Volvo also started in spring 2018, as Volvo outlined the need for an XR headset that would allow them to test various elements of future cars – such as heads up displays, new materials, and UI for infotainment systems – inside a real car while driving on a real test track. The high requirements on readability and low-latency needed to drive a car on a test-track pushed Varjo to succeed in product development.
Given how well Unity already worked for the VR-1, it was a natural choice to try out how the virtual objects would appear in mixed reality. The fact that Unity is easy to integrate and extend with C++ libraries, such as our own Varjo plug-in, made it possible for us to extend our plug-in to support mixed reality. By simply defining the empty background in a VR scene to be replaced by the video-pass-through signal, we were quickly able to see virtual objects in a real environment.
The close collaboration and fast iterations were made possible by Unity’s ease of use, as our team was developing and improving pass-through simultaneously while working hand in hand with our customer. A year later, the first public demonstration of XR-1 brought to life the capabilities of our technology, combined with Volvo’s superior models and photorealistic Unity graphics.
The demo illustrates the power of video pass-through mixed reality as opposed to optical see-through. In this demo, you have the following steps:
1. Experience real reality
You see the real world around you through the XR-1 headset. The real world is streamed with <10ms latency via the high-res cameras in the front plate. You see the world in a full field of view and at a high resolution with a 90Hz framerate, which gives a sensation of not wearing any headset at all (i.e., seeing the real world with your own eyes). You can walk around and explore the real world freely.
2. Enter photorealistic mixed reality
A beautiful Volvo XC60 builds up in front of you. It first appears as a stylized transparent blue wireframe. The virtual car is anchored to the real floor in the room around you and oriented so that the chair in the booth is aligned with the driver’s seat of the virtual car. The viewer can take a seat in the real chair and is still able to see the real surroundings through the wireframe.
The car now turns into a solid model, and the surfaces goes from transparent to opaque. The virtual car casts shadows on the floor of the real world, and looking on the car’s surface, it is possible to see that the real world is reflected in the car’s surfaces. The reflections come from an HDR cube map that was taken during setup on the exact spot of the car. The same cube map is also used for ambient lighting.
This is the first time the viewer sees opaque mixed reality, and the effect is stunning. You can still see the real world and your colleagues through windscreens.
How it was accomplished: The car model was provided to Varjo by Volvo. Because the resolution of the headset and the car was so high, we needed to do as much pre-processing as possible. The lighting was baked in the DCC to textures and multiplied in custom shaders. The baked textures only dealt with occlusion and the shading is still affected by the skybox.
Mattias Wilkenmalm from Volvo handled asset creation and wrote custom car paint shaders that delivered superior results. We simply modified them to get the look and transitions we needed. The final model is around 7 million polygons and has around 150 4K textures.
3. Switch seamlessly into virtual reality – and back.
The viewer is then asked to step outside the car, and we transition to Venice. The last pieces of the real world around the viewer are now disappearing in a unique transition as the reality transforms into a virtual scene of Venice, where the car is parked in one of the alleys. The reflections in the car are now those of Venice, and the shadows of the car are now landing on the streets of Venice.
After a while, we transition back from virtual to the real world. The user can now go around the virtual car and see all the details and reflections. This shows that XR-1 offers the ability to still interact with others and select only the parts you want to virtualize.
How it was accomplished: To make the transitions visually pleasing, Volvo’s Timotei Ghiurau, Lead, Virtual Experiences, and XR Research, used world space 3D noise with alpha cutouts to bring in the car and the environment. This is fast to do in the fragment and it looks very cool. It was a perfect combination when dealing with tight deadlines. Noise functions can be fetched from Unity’s Kejiro’s repository.
To get the smooth transition for the car reflections, the Venice environment was added to a separate layer so that the real-time reflection probe was only rendering minimal amount of geometry. The reflection probe was rendered at 30 frames per second while the scene renders at a much higher frame rate. Having the transition visible in the reflection probe added much more immersion to the scene.
The fact that XR-1 is a first-of-a-kind headset to offer the ability to seamlessly switch from real reality into mixed reality onwards into full VR and back to reality again makes for a very impressive demo. It is a Matrix-like moment to see the surrounding reality disappear and replaced by a virtual scene – and then traveling back.