Crossing into the 4th dimension – how Carl Emil Carlsen creates generative art with Unity

Have you experienced visual music or witnessed a virtual performer dance right into the audience? In this post, we go behind the scenes with an innovative Danish creator who challenges viewers’ perceptions by projecting Unity-generated experiences in theaters and exhibition spaces.

From his lab in the crypt of an old church in Copenhagen, Carl Emil Carlsen – an interaction designer and media artist – shared why he likes to release “living” digital forms from the confines of conventional devices. 

“I’m interested in crafting interactive audiovisual illusions that merge virtual and physical phenomena. When I make digital entities escape the screen, I like to think they are just like early lifeforms escaping the ocean. It's exciting to shape this hybrid space as technologies evolve – there are so many new paths for artists to explore.”

A clip from Silicium2, Carl Emil’s project with techno artist Bjørn Svin

A big part of Carl Emil’s approach is how he uses technology to achieve his artistic goals. “Tools shape our thinking, so I want to have some influence over them. Unity lets me do that because it’s highly customizable and I can easily experiment with many different formats, media, and types of content.” 

And he’s not doing this in a vacuum. “Throughout my career I’ve tried to choose projects where I’m going to learn something. I try to find challenges that will take me somewhere new. And on that path, I’ve been lucky to collaborate with some very skilled people, who’ve taught me a lot.”

Some of Carl Emil’s tools for creating “gen art,” including strong coffee

In a Unite Copenhagen talk, Carl Emil points out how his interest in mixing physical and virtual spaces led him to explore the 4D Box, a live stage located at The Culture Yard in Helsingør, Denmark. “What excited me about it was the possibility of producing hologram-like illusions for a large audience.” 

The stage is based on an optical illusion popularized over 150 years ago. John Henry Pepper mesmerized London theater-goers by reflecting hidden actors onto a theatre stage using a large piece of glass and a lamp, making them appear like ghosts. Consequently, the technique is referred to as “Pepper’s Ghost.”

The 4D Box extends this technique with a passive stereoscopic 3D system, which means the audience (wearing 3D glasses) can experience virtual props and scenography. To test viewing angles and content scaling for the project, Carl Emil made the Pepper’s Ghost model below (available on GitHub).

Carl Emil’s Pepper’s Ghost mockup for the 4D Box stage

“To create the best-possible 3D perspective, I overwrote Unity’s camera projection matrix with off-axis projection using OffAxisCamera from the Asset Store. I also extended Unity with an ArtNet DMX implementation to take control of the stage lights and match the virtual lighting with physical lighting.” 

To virtualize the performer’s body, Carl Emil uses a Microsoft Kinect sensor, which tracks the body’s movements. “I need to calibrate the tracking with the projection so that virtual elements can be aligned perfectly with the body. Then I use the depth image from the sensor to mask out elements that move behind the actor.” To synchronize everything, he handles the animation in Playhead, a custom tool he built in Unity, based on Unity UI and OSC simpl from the Asset Store.

Carl Emil’s custom Playhead tool in action

His first experiment in the format was Seasonal Skin, a runner-up in the Unity Awards’ Best Non-Game category some years ago. He used the stage as an interactive installation on which the audience could experience being dressed in virtual costumes. Since then, he has done a wide range of 4D Box productions, experimenting with new technologies and Unity features as they emerged.

Seasonal Skin: An early blending of generative art and a live performer. Carl Emil has been using Unity since 2006 (version 1.4).

Not long after the Seasonal Skin project, Carl Emil started the visual music duo Silicium with Danish techno artist Bjørn Svin. Together, they focus on creating audiovisual and synthetic-organic interactions in live performances. So far they’ve created four 4D Box productions, with the latest one shown at Ars Electronica in Linz, Austria. According to Carl Emil, “Starting with a core concept, we develop real-time synthesized material in parallel, working towards a coherent audiovisual universe, an approach similar in some ways to game design.”

Last year, Silicium produced “Primordial,” a visual music concert in a tour-friendly format based on a cinematic 2D setup. This production premiered at CPH:DOX and was most recently shown at Art & Tech Days in Košice, Slovakia.

Gen art from Silicium’s Primordial project

Carl Emil handles most of the animation in Primordial via Playhead, while some parameters are controlled live through Durga, a multi-touch GUI that he also made with Unity. Durga allows him to blend influences from the music and various random behaviors in an improvised manner. The virtual camera is controlled freehand by a 3DConnexion Space Navigator, integrated using the free SpaceNavigator Driver from the Asset Store. 

To orchestrate the gen art that’s at the core of his experimental work, Carl Emil has developed a specialized graph framework in Unity. This primarily routes ComputeBuffers between various ComputeShader passes. The graph UI is built in UIElements and is an example of a specialized graph tool that Unity users can build for their own needs.

The setup for Primordial’s visual simulations shown in Unity 2019.3

On the commercial front, Carl Emil also exercises his experimental chops. Approached by Perstorp, a Swedish company promoting a product to reduce the amount of antibiotics used for meat production, he led the design on an interactive expo to draw awareness to drug-resistant bacteria.

“To do that, I got a single machine running Unity to render to six HD projections, five of which covered a typical VR cave setup. I did the rendering with the High Definition Render Pipeline (HDRP), with non-interactive parts played back as HAP codec video using AVPro Video from the Asset Store.” 

He built a Swedish forest environment, largely based on the free Book of the Dead assets by Unity’s Demo team, which served as background for illustrations by Sidsel Sørensen and Maya SB. Narrative and sound design was produced by Simon Kirk, while the hardware setup was designed by Vertigo, a light art studio that hosts Carl Emil’s workspace.

He pointed the sixth projector at the audience for a special surprise. At the height of the experience, the virtual bacteria crossed the screen-space border to enter the human realm, reminding the audience how such issues can become a personal matter.

Scenes from the experience, including “live” virtual bacteria

Besides his “screenless” work, Carl Emil has brought his background in interactive design and media to a number of interesting small-screen projects. “I love getting involved in the early stages of concept development, prototyping interactive elements to explore what the product may feel like.” 

For example, at ustwo (makers of Monument Valley) he was involved in AR prototyping for LEGO and Hasbro. The Hasbro prototype was later turned into a toy product called Play-Doh Touch. For the prototype, he procedurally transformed images of plain Play-Doh into rigged 3D characters using the OpenCV integration from the Asset Store.

From Play-Doh to rigged AR prototype

“I’m continuing work with The Culture Yard. They’re currently producing a new 4D Box theater performance about personal data and artificial intelligence (AI), which will preview at the upcoming Click Festival.” 

For this, Carl Emil is developing interactive graphics, including a visualization of a chat-bot AI created by artist Cecilie Falkenstrøm and her team. “The AI will converse with an actor and the audience as a non-linear narrative unfolds, providing insight into how our data is being harvested and used, which is mostly hidden from us. I hope this piece will leave the audience with a poetic and sensory awareness of the issue.”

In April, Carl Emil is going to Taiwan to tour with Chronicle of Lightyear, a cross-cultural 4D Box production by Very Theatre (TW) and the Culture Yard.

If you’re interested in generative art/graphics, Carl Emil says that a good place to start is the tutorials by Catlike Coding (on Patreon). You can also check out examples by our Tokyo-based evangelist Keijiro Takahashi. To explore what else is buzzing in this corner of the Unity community, join the Audiovisual Unity Facebook group that Carl Emil moderates. You can also follow Carl Emil’s work on Twitter, and check out more projects at his website.