David Bowie is in AR: How Planeta built a killer app with Unity
«David Bowie is» is an app created by creative sound and media studio Planeta. Planeta used Unity to adapt the original museum exhibition for AR, featuring high-resolution captures and immersive settings. Here, Planeta talk about some of their creative and technical challenges.
The music world lost a true original with the passing of David Bowie in 2016. Bowie was more than an iconic pop star; he was a consummate artist and innovator, continually reinventing himself with new stage personas and moving through musical styles and genres with ease. With a career that spanned 40 years, David Bowie cast a long shadow on world culture.
Bowie was wild about technology and how it affects the human experience. His first big hit, “Space Oddity,” released before the Apollo moon landing, told the tale of a wayward astronaut longing for home. His 1976 song TVC15 warned about the addictive nature of television via a “hologramic” broadcast – a full year before Princess Leia projected onto a tabletop. And in the early days of the Internet boom, Bowie saw the disruptive potential of the web and started his own AOL-style ISP, BowieNet, to serve his fan community.
The exhibit David Bowie is premiered at the Victoria and Albert Museum in London in 2013, honoring the artist while he was still alive, and traveled the world for five years. As the tour was winding down, leads at Sony Music began thinking about how to preserve the essential elements of the show – and Bowie’s legacy – in XR, and found the team at Planeta to help create an AR experience.
The result is the app David Bowie is, an augmented reality recreation of the museum show that features 2D and 3D representations of artifacts from the show, along with hours of audio and video of Bowie’s music and performances. The app was made with Unity, using our ARKit support for the original iOS version, then moving to AR Foundation to build for Android. The app was launched on Bowie’s birthday, January 8, 2019; he would have been 72. It’s awesome. As an avid Bowie fan myself, I think he would have loved it.
I recently caught up with the folks from Planeta – Nick Dangerfield, Jimi Stine, Dan Brewster and Pilar Aranda – to talk about the project:
How did this project come to land at Planeta?
Akiko Ozawa, of Sony Music Entertainment (Japan) Inc., is a devoted fan of David Bowie and of the V&A show. Two years ago, she brought David Bowie is to Tokyo; when the show did its final leg in Brooklyn last year, Akiko thought that a full copy of the show should be preserved in XR for posterity. So she started looking for a studio to help out. As our work on immersive tech is mostly around music and visual arts, Akiko felt that we’d be up to the task.
What were your biggest challenges on this project both technically and creatively?
At the outset, our biggest challenge was answering the questions: What does a mobile AR gallery look like? How do you move through it? We’re no strangers to creating virtual art galleries – Planeta has developed seven VR gallery/exhibition spaces since 2016 — but to do so for the small screen, while simultaneously incorporating over 400 objects of varied media, required days and days of design discussion to figure out. The technical questions primarily dealt with the massive amount of asset data. Discerning the most coherent and beautiful way to display these objects was one thing, but doing so smoothly and seamlessly was a challenge all its own. In that sense the design was limited by the tech – there’s only so much that mobile AR can handle. Making sure that every item was properly tagged with an accurate description and title also took a considerable amount of QA.
Describe how Unity helped make your process easier or solved a problem for you. Were there any specific Unity features you leveraged?
AR integration via AR Foundation was great to get us up and running. As a tool, the overall malleability of the Editor was extremely useful in creating specific workflows that saved us countless hours. The asset store ecosystem also saved us tons of time as we were able to purchase tools and visual effects that would have taken up far too much of our limited resources otherwise.
What artists/creators have inspired your work and to do what you do?
This will definitely sound corny, but in this case our guiding light was David Bowie and his constellation of influences, which is essentially saying the best art made in the twentieth century. David Bowie had an incredibly fine eye for good work, and one of the perks of this project has been revisiting some of the most influential artists of our time, from Bauhaus to Brian Eno.
What advice would you give someone embarking on a similar archival-type project in AR?
It’s important to understand that David Bowie is acts as a recreation of an expansive exhibition on a phone more than it does as an archive. That being said, it’s really important to capture the display materials at the best possible quality. The images we made for the project were taken by an archival art photographer from the Metropolitan Museum of Art.
As for advice related to future galleries, we’d say it’s important to remember that we are no longer confined to the architecture of traditional museums. While it’s understandable that other projects seek to replicate the museum experience 1:1, we feel projects are more successful when imaginative spaces are created, leveraging only certain techniques of standard presentation to help acclimate the user to the experience. So much can be done with sound and light to create an optimal or interesting atmosphere in which to view a piece of art – why limit ourselves to the same white-walled incandescent light-filled boxes we’ve always known?
How did you optimize all the object scans to render so well in mobile AR? What photogrammetry software did you use?
The models were the most labor-intensive aspect of the whole project. After photographing and scanning each costume (for the record, that’s over 50 costumes captured in five days!), we used a combination of Agisoft Photoscan and Maya to combine all our data. From there, the finishing was all done by hand and was completed only a few days before we launched.
To optimize the assets we did tons of balancing and play-testing to test for performance and overall usability. Ultimately, we decided, in most cases, to show a reduced poly model in the scene context, and up-res it after it had been selected. Similarly, all the models are presented without lighting data once they’ve been selected, which helps to show off the detail captured in the photogrammetry data.
I love how each scene starts by rendering the pass-through camera and eventually fades to black, making you feel like you could be in the museum. Was this purely for aesthetic reasons or was performance optimization a factor?
Fading to black was purely aesthetic. We played around with varying levels of background opacity for the duration of development; at one point we almost included a slider that would let users adjust the background opacity as much as they wanted. Ultimately, however, we decided to keep the background black to keep the attention on the assets themselves. Too much camera view would lead to confusion between real and virtual spaces. While that blurred line is usually considered a perk of AR, we felt that if we were going to create an immersive, museum-quality experience, we would need to have total control of the visuals.
How do you feel about the end result? Is it the end?
We are very happy with the end result, especially given the limited time frame we had to work with (roughly six months from start to finish). There are, of course, features we would have loved to include, but there just wasn’t the time to implement them. As of right now, we don’t have any plans to continue working on David Bowie is.
What’s next for Planeta?
Our next digital project will build extensively on our history of making new kinds of spaces to exhibit more traditional forms of art. Incorporating what we’ve learned from our past work, and partnering with artists and institutions alike, we aim to create a new cultural events space in virtual reality. This project would be a multiplayer environment with both permanent and temporary exhibitions, designed alongside the best minds in architecture, sound design and lighting.