For the last four years, Belgrade, Serbia, in the middle of April has been the place to meet for new media artists and creative coders from all over the world. Resonate Festival features more than 50 lectures and 20 workshops. During the six days of the, festival dozens of artists showed their projects ranging from pure analog installations to very sophisticated combinations of hardware and image processing algorithms.
It is really awesome that with the release of Unity 5 Personal Edition more and more artists and creative coders working with 3D and especially VR started using Unity to prototype their ideas.
One of the 3-day workshops at the festival was organized by James George and Alexander Porter. The participants used Unity to explore 3d-scanned areas of Belgrade. They used the photogrammetry technique to scan buildings and other objects, imported them into Unity 5, added a first-person controller and walked around semi-real virtual spaces.
How cool is that!?
So, I decided to ask the guys to tell us more about this workshop.
Valentin: First of all, could you please tell the readers about yourself and what you do?
James: I’m an artist who works with computational photography and interactivity. I studied computer science, but have been continually drawn towards the more expressive and creative side of the sciences. I think there is a lot of unexplored potential for emerging vision and graphics technology to get at a deeper representation of people and their environments. I want the art forms traditionally associated with analog mediums like photography and cinema to be realized within interactive, virtual space.
Alexander: I’m an artist with a background as a photographer and cinematographer. Increasingly, I’ve been adding 3D practices and computational photography to the way I create imagery. My work often leads me to combining traditional camera technique and scanning workflows. I have a particular passion for documentary work – exploring new ways of capturing people and the world for telling our stories.
Together with a third partner Mei-Ling Wong, James and I founded a studio called Specular. At Specular, we do both commercial projects involving 3D scanning, game engines or interactivity and meanwhile we run workshops and art residencies to explore ideas and techniques like the ones behind Exquisite City.
V: What was the idea behind the workshop?
A: We were inspired by the surrealist drawing exercise of the Exquisite Corpse, where a group of people collectively draw a human figure without seeing the contributions of their collaborators. The exercise results in surreal images of disjointed and fantastical beings. We wanted to expand that exercise to the scale of a city. The Exquisite Corpse concept combined with the research I’ve been doing into photogrammetry scanning resulted in the idea for the workshop as a collaborative project to scan, segment and remix the architecture of the neighborhood surrounding our workshop in downtown Belgrade.
During the first day, we taught the photogrammetry workflow and sent all the participants out to wander the streets of the city to capture parts of the built environment. They came back with a wonderful variety of surfaces, objects and building facades. On the second day, we processed all the geometry and isolated certain parts to create modular prefabs that would allow us to use the parts as building blocks.
On the third day everyone contributed the prefabs they’d created to a pool of models and we assigned them city blocks on the map of the neighborhood and we began to build. The result is a surreal and beautiful urban landscape that still seems to keep those familiar urban forms, textures and sounds intact.
J: A lot of the workshops taught at Resonate required a lot of prerequisite knowledge about coding and design. Our hope with the Exquisite City was to teach new techniques, but ones that were accessible to a diverse group of participants. We didn’t want to require any programming to make our city, yet provide a framework for advanced users to dig deeper. Unity’s drag and drop interface was perfect for this, it allowed the participants to freely arrange their city and use predefined scripts to move through it.
V: How exactly were you capturing the buildings?
A: We used a technique called photogrammetry to create 3D models of specific portions of the city. The workflow required us to take a series of photos of an object – a street corner, a bench or even small architectural features on a facade – and based on computing the differences between the images, a photogrammetry tool enables us to build a very detailed 3D model of the object.
Photogrammetry can be a powerful tool for 3D scanning, but it notoriously creates large and chaotic meshes. To circumvent those issues and to save time in the workshop, we decimated the meshes quite severely and exported the detailed photographic textures to retain some of the beautiful and eccentric surfaces of the city-scape.
J: Most participants were surprised that a process like photogrammetry was even possible, let alone something they could do themselves with their smartphone cameras. There is a real magic to seeing a collection of photographs transform into a model that can be interacted with in a virtual world.
V: What equipment do you need to do this?
A: One of the interesting things that distinguishes photogrammetry as a technique from other 3D scanning methods (LIDAR, structured light etc.) is that it has very simple hardware requirements. You only need a camera and in fact a few of the participants in the workshop used camera phones with great success. We used Photoscan, which is a paid tool with a free full-featured trial, but there are free and/or open source alternatives out there — 123D Catch from Autodesk, also two open source toolkits; VisualSFM and a Python photogrammetry toolkit.
V: And when you got a model you just drag it into Unity editor, right?
A: Yes, the models can be exported right from Photoscan into Unity as an FBX. However, in a less beginner-oriented workflow one would definitely spend a fair amount of time refining and remeshing models in a mesh modeler before importing it to Unity.
J: The immediacy of these tools is what is most impressive to us. The fast that this very technical and emerging process already has a suite of What You See is What You Get workflows really enables creativity and lowers the barrier to entry.
V: How experienced were the participants with Unity?
A: With a couple of exceptions they hadn’t used Unity at all before the workshop. It ended up being a perfect sandbox and teaching the interface turned out to be around half as difficult as the other tools in the workflow, which allowed us to get into the process of laying out our strange city quite quickly.
V: Did you provide template prefabs for the participants to use?
A: We thought about the overall shape of a city and came up with ways to separate it into discrete units. We came up with a set of typologies that would allow us to recombine them in arbitrary and interesting ways, using them almost as 3D building blocks. We’ve put the templates and tutorials for the whole workshop up on our workshop page.
J: We took a page from the exquisite corpse drawing game on this one. There needed to be defined boundaries for each person to create within. This had the added benefit of another level of absurdity, as small scans of plants or doorways scaled to the same size as statues and entire facades.
We also provided the visitors with a city street template and assigned separate blocks to each participant. As you explore the final result, you can see that each city block you enter was crafted by a different creative mind and has a unique style.
V: This is not your first Resonate, right? What do you like about the festival?
A: It is not, we attended the first Resonate festival in 2012 and taught a workshop with our tool DepthKit, a volumetric filmmaking tool for filming with the Kinect or other depth sensors. As a side note, we are actually in the process of converting it into a robust tool for capturing and viewing cinematic 3D video captures in Unity for use especially in VR and AR.
The festival has an incredible energy about it. We get the chance to meet and get to know so many friends and colleagues with whom we are usually only in virtual contact. The quality of the workshops and talks and the passion of the organizers is really inspiring.
J: Resonate provides a one of a kind audience and community for us to bring these types of workshops. We’re incredibly grateful to the organizers Eduard Prats Molner and Julie Pusztai for giving us the opportunity to create the Exquisite City.
Christian Ervin, Pierluigi Dalla Rosa & Paul Skinner of Tellart lead a physical computing workshop “Designing Connected Experiences”.
Project templates and tutorials
If you want to explore the final project please visit the Project page where you can download executables for Windows and Mac (+Oculus Rift support!). All Unity templates and tutorials are available for free at the Workshop Page.
So, don’t waste your time, take your iPhone and start making a first-person shooter out of your apartment or office space! (8
If you want to find out more about the festival follow its page on Facebook where you will find photos and reviews of past events, like this one from dailyinspiration.nl — Resonate 2015: For Those of You Who Couldn’t Attend. If you’d like to play the game created during the workshop you can do so here.