This is a guest blog post
Hi, I’m René from IdentityMine/Valorem and a passionate developer working with 3D computer graphics. I’ve been in VR and AR for many years now. With all the amazing VR and AR/MR devices coming out, it’s a great time to be a developer working in that immersive space using Unity. As part of Microsoft’s HoloLens Agency Readiness Program, I was lucky enough to have been working on HoloLens experiences since last year and in this post I’d like to share some of the stuff we’ve done and things we learned.
My team and I are developing lots of different HoloLens apps and games for many different use cases. I can’t share all the projects we are working on but this video and the screenshots below give you a glimpse of the things we have been working on.
We also recently released a little game for HoloLens called Tower Blocks. Tower Blocks was actually created by experimenting with two-handed interactions where we tried out new ways to transform holograms. We had so much fun playing it that we released it for free in the Microsoft Store.
I also shared what I’ve learned in recent talks presented and recorded at Unity’s Unite Europe event and Microsoft’s //build conference as well as a few blog posts like Getting Started with HoloLens Development and Top 10 HoloLens Development Recommendations.
A good portion of the talks and posts covered best practices and other things I learned while working on a project called HoloFlight which is a near real-time flight data visualization for HoloLens using real-world topographic and flight data showing the data in 3D as holograms instead of flat 2D plots.
The HoloLens Emulator
At IdentityMine, we had access to some HoloLens devices but not all of our team had access to them, therefore most of HoloFlight was actually developed using the HoloLens emulator and the special Unity build for HoloLens, which everyone can download for free.
HoloLens is a Windows 10 device and the HoloLens emulator runs as a Hyper-V virtual machine. You can learn more about system requirements here. The human input like gestures and other sensors are simulated using the PC’s keyboard, mouse, or Xbox controller.
It is really easy to deploy a Unity game / app to the emulator. It just has to be built like a regular Unity app for HoloLens. Then from the Visual Studio output solution the right Platform and Target Device have to be selected from the dropdown:
The emulator can simulate positional changes (user walking), as well as rotational changes (user head rotating), gestures, hands and even voice commands.
Position and Rotation
The rotation can be simulated by pressing the left mouse button and moving the mouse. Positional changes can be simulated using the keyboard’s W, S, A, D and Page Up/Down keys.
Gestures and Hands
The air-tap gesture can be simulated with a right mouse button click; if the Alt key is pressed while pressing the right mouse button, the tap+hold gesture for hand scrolling movement can be simulated. The built-in bloom gestures which brings up the Windows start menu is mapped to the keyboard Windows key.
Voice and Speech Recognition
HoloLens is a Windows 10 device so it supports Microsoft’s voice-controlled assistant Cortana for speech recognition. This works well with the HoloLens emulator.
Hand and head rotation and much more emulation features can be controlled even further. A detailed list of all simulated emulator inputs can be found here.
Besides the fact that HoloLens is a fully untethered device, which can render holograms, another unique feature is that it constantly scans surroundings and performs a surface reconstruction providing a spatial map of the environment to your application. This feature alone is very impressive as you can see in this demo video when the virtual balls roll down in the real garden.
Of course the emulator also supports spatial mapping and, even better, it can actually load different environments / rooms to test.
These rooms can be extracted from a HoloLens device as .xef files, so you could use a HoloLens device, scan an environment, save it and then load into the emulator. For example the screenshot below shows a spatial mesh of a real-world staircase loaded into the emulator:
Which is similar to this when seen through the HoloLens device:
More details about the emulator and its Room feature can be found here.
The HoloLens toolset provides you with everything you need to get started for HoloLens development, including an amazing emulator, which is close to the real device. All of these tools are free of charge! Of course nothing comes as close to the behavior of a real device and in the end it is needed for a professional app to tune performance and UX but the emulator can provide a very good starting point. We leveraged the emulator in a distributed team to speed up development and to not be constrained by device availability, which worked out very well for us.
By the way, if you are interested in working with VR, AR and real-world HoloLens development, IdentityMine and Valorem are always looking for talented developers, designers and artists.
About author: René Schulte is the Director of Immersive Experiences at IdentityMine / Valorem Consulting. He has experience with VR, AR, MR development and 3D programming for more than 10 years and has been developing for HoloLens since 2015.
René Schulte also created and maintains a few popular Windows open source libraries like WriteableBitmapEx or the Augmented Reality library SLARToolKit. For all his community work, René Schulte was honored with the Microsoft MVP Award.