Search Unity

Stereo 360 image and video capture

January 26, 2018 in Engine & platform | 4 min. read
Topics covered
Share

Is this article helpful for you?

Thank you for your feedback!

We are proud to announce that in 2018.1 creators can now capture stereoscopic 360 images and video in Unity. Whether you’re a VR developer who wants to make a 360 trailer to show off your experience or a director who wants to make an engaging cinematic short film, Unity’s new capture technology empowers you to share your immersive experience with an audience of millions on platforms such as YouTube, Within, Jaunt, Facebook 360, or Steam 360 Video. Download the beta version of Unity 2018.1 today to start capturing.

How to use this Feature

Our device independent stereo 360 capture technique is based on Google’s Omni-directional Stereo (ODS) technology using stereo cubemap rendering. We support rendering to stereo cubemaps natively in Unity’s graphics pipeline on both Editor and on PC standalone player. After stereo cubemaps are generated, we can convert these cubemaps to stereo equirectangular maps which is a projection format used by 360 video players.

To capture a scene in Editor or standalone player is as simple as calling Camera.RenderToCubemap()  once per eye:

          camera.stereoSeparation = 0.064; // Eye separation (IPD) of 64mm.

           camera.RenderToCubemap(cubemapLeftEye, 63, Camera.MonoOrStereoscopicEye.Left);

           camera.RenderToCubemap(cubemapRightEye, 63, Camera.MonoOrStereoscopicEye.Right);

During capture of each eye, we enable a shader keyword which warps each vertex position in the scene according to a shader ODSOffset() function which does the per eye projection and offset.

Stereo 360 capture works in forward and deferred lighting pipelines, with screen space and cubemap shadows, skybox, MSAA, HDR and the new post processing stack. For more info, see our new stereo 360 capture API.

To convert cubemaps to stereo equirectangular maps, call RenderTexture.ConvertToEquirect() :

           cubemapLeftEye.ConvertToEquirect(equirect, Camera.MonoOrStereoscopicEye.Left);

            cubemapRightEye.ConvertToEquirect(equirect,Camera.MonoOrStereoscopicEye.Right);

Using Unity frame recorder, a sequence of these equirect images can be captured out as frames of a stereo 360 video. This video can then be posted on video websites that support 360 playback, or can be used inside your app using Unity’s 360 video playback introduced in 2017.3.

For the PC standalone player, you need to enable the “360 Stereo Capture" option in your build (see below) so that Unity generates 360 capture enabled shader variants which are disabled by default in normal player builds.

In practice, most of 360 capture work can be done on the PC in Editor/Play mode.

For VR applications, we recommend disabling VR in Editor when capturing 360 stereo cubemaps (our stereo 360 capture method doesn’t require VR hardware). This will speedup performance without affecting the captured results.

Technical Notes on Stereo 360 Capture

For those of you using your own shaders or implementing your own shadowing algorithms, here are some additional notes to help you integrate with Unity stereo 360 capture.

We added an optional shader keyword: STEREO_CUBEMAP_RENDER. When enabled, this keyword will modify UnityObjectToClipPos()  to include the additional shader code to transform positions with ODSOffset()  function (see UnityShaderUtilities file in 2018.1). The keyword will also let engine setup the proper stereo 360 capture rendering.

If you are implementing screen space shadows, there is the additional issue that the shadow computation of reconstructed world space from depth map (which has post ODS Offset applied) and view ray is not the original world space position. This will affect shadow lookup in light space which expects the true world position. The view ray is also based on the original camera and not in ODS space.

One way to solve this is to render the scene to create a one-to-one mapping of world positions with screen space shadow map and write out the world positions (unmodified by ODS offset) into a float texture. This map is used as true world positions to lookup shadow from light space. You can also use 16-bit float texture if you know the scene fits within 16-bit float precision based on scene center and world bounds.

We’d love to see what you’re creating. Share links to your 360 videos on Unity Connect or tweet with #madewithunity. Also, remember this feature is experimental. Please give us your feedback and engage with us on our 2018.1 beta forum.

January 26, 2018 in Engine & platform | 4 min. read

Is this article helpful for you?

Thank you for your feedback!

Topics covered