Search Unity

In this third and final part of the series we analyze how foundational platforms will and must evolve, and outline our vision of how MR applications will be created and executed. Part I summarized the current and future issues facing mixed reality and its adoption in the mainstream. Part II explored the design challenges facing developers with this new mixed reality medium.  

Foundational MR services

The tools and services Unity provides for creating apps -and the apps themselves- must live in and interact with a set of foundational services. So in order to understand the future of MR creation/adoption and where we should go, we take a step back and think holistically.

Facebook’s purchase of Oculus back in 2014 kicked off  the latest wave of consumer virtual reality devices and applications, and because of it, mixed reality has been predicted to become the next computing platform. Goldman Sachs stated in early 2016 that VR and AR have the potential “to emerge from specific use cases to a broader computing platform”. A March 2017 Forbes article titled “The Next Mobile Computing Platform: A Pair Of Sunglasses” projected that MR glasses will replace mobile phones, monitors and TVs, with foundation software and services evolving accordingly.

Articles like this one from Reuters  characterize this emerging platform as a combination of augmented reality, artificial intelligence and cloud-based services, and we see major tech companies offering each their own perspective and implementation. Clay Bavor sees the combination of VR and AR with artificial intelligence and machine learning as enabling “the next phase of Google’s mission to organize the world’s information”. Through their Windows 10 Fall Creators Update and VR/AR devices, Microsoft aims to build “the platform that empowers everyone to create.” In June this year Apple introduced ARKit with iOS 11, and their keynote presented the A11 Bionic chip with its neural engine for performing artificial intelligence, initially aimed at facial recognition.

Operating systems and platform services are already evolving to include features for immersive technologies. Android Nougat introduced support and optimizations for virtual reality at the OS level. Windows Mixed Reality Platform APIs are part of the Universal Windows Platform. And as we saw in this year’s keynote, Apple’s new hardware is built for AR and AI, with the operating system and SDKs tightly coupled for optimized use. Now, what else do we need in a MR OS? What does it look like?

A paper from the 14th Workshop on Hot Topics in Operating Systems explored how operating systems should evolve to support AR applications. Since natural user input is needed for AR, the authors reimagine input within the context of it being  a continuous sensing of the real world. They analyse user privacy mechanisms that should be put in place due to sensitive info mixed in with raw data. Furthermore, they argue sensor input access should not be restricted to one application at a time; instead it is desirable to let multiple AR apps from different vendors simultaneously read sensor inputs and render virtual overlays in a shared 3D reality. They also point out how the synthetic window abstraction in traditional GUIs is no longer viable so OSs must evolve to “expose 3D objects as the display abstraction and perform display isolation and management in the 3D space.” And functionality that is very common across AR apps, such as computer vision and AI, should be offloaded from applications to dedicated OS modules.  

Now, as described in our previous articles, we are re-thinking the principle that “one content or service equals one application icon”. We believe MR should be human and object centric, and use the world as a device (aka “clickable world”).  So we are re-examining the notion that applications are presented as separate icons that must be clicked on by the user in order to be executed. In this new computing platform where computer vision and AI are foundational services, we envision MR experiences launching not just when requested by the user but when they are needed (e.g. a battle game automagically starting when user picks up a physical nerf gun, or microwave cooking instructions appearing when user picks a dinner meal from the fridge). This would require OS/platform services that allow apps to register their launching conditions, an extension of automating location-based tasks on mobile devices.  

In previous articles we also mentioned our projection that sensors outside MR devices will populate physical spaces around us. The IoT will communicate bidirectionally with MR wearables. Clearly this scenario is inspired by Vernor Vinge’s Rainbows End – “Cryptic machines were everywhere nowadays. They lurked in walls, nestled in trees, even littered the lawns. They worked silently, almost invisibly, twenty-four hours a day”–  but there are already examples of general purpose sensing, like the Synthetic Sensors initiative at Carnegie Mellon University.  Sensors are needed to allow better understanding of the physical world, but as Greg alluded to in part II, the world as a playground and keeping users connected to the tangible world around them also requires the ability to manipulate physical objects from within mixed reality. Some current examples that allow this bidirectional communication are the Reality Editor and Open Hybrid. What this means for the emerging MR computing platform is that OS services on MR glasses should extend the concept of input/output devices to include new non-local hardware that it identifies, connects to and possibly controls at run time (i.e. recognize and connect to environmental sensors when entering a room, or identify and interact at run time with smart outdoor cameras) so that individual applications don’t have to reimplement this every time.

So, based on current trends and how we believe the “next computing platform” should evolve, these are some of the  assumptions for our research work:

  1. Connectivity will not be an issue: Fifth-generation (5G) wireless networks are predicted to compete with fixed broadband. Light Fidelity (LiFi) communication has been deployed and is expected to take off soon in the US and Europe, perhaps also in the rest of the world.
  2. Operating systems and platforms will include machine learning and computer vision services (some of them running in the cloud), and allow bidirectional communication with the IoT.
  3. Cloud services will allow storing a persistent matrix, i.e. a digital copy of the physical world, and the synchronized shared mixed realities. This way, devices become windows into persistent realities.
  4. A variety of sensors and devices connected to the IoT will populate environments and  communicate bidirectionally with MR devices.

Of course we don’t really know exactly how services and technology around these assumptions will shape up and the profound implications on what an engine like Unity is able to build as features. Additionally, data privacy and security implications are large and need to be considered deeply by everyone involved.  We hope our experimentations bring some answers and shape the field towards mixed reality becoming mainstream the right way.

Augmenting applications

In a world described in the previous section, we envision MR experiences taking the form of contextual applications like the business card and the cereal box game presented in part II of our blog. These types of applications would require shifting some paradigms around creation and development.

First, we address launching MR experiences when needed. As part of the process of creating an MR app in Unity, creators should be able to specify the conditions that must be met to launch the app. We use the term contextual trigger for this, and define it this way:

Contextual triggers are the criteria around a user’s virtual or physical environment that activate an MR experience. They are a combination of time of day, GPS coordinates, proximity to certain physical object(s), fiducial markers, serial number on an internet-enabled device, and through brain-computer interfaces and advances in neuroscience and machine learning,  contextual triggers will come from users’ intent, preferences and behaviors. The contextual trigger for a specific experience will be the set of predicates that when valid will cause an MR experience to start.

Second, we address moving away from applications as isolated executables that run on their own memory space. Instead, we define a contextual application as a collection of data that defines an MR experience and has access to a persistent reality. In simple terms, we “augment” the classic definition of interactive app to be composed of:

  • Contextual triggers
  • Application logic, aka interaction rules: how it interacts with the user and the world
  • Assets: 3D/2D visuals and audio that are rendered on the client.

Contextual triggers and apps are just groundwork concepts for upcoming research and experimentation. We are currently elaborating on them and prototyping, but there are many more questions to answer, e.g. when 3D objects (instead of windows) are the display abstraction, what should the Unity canvas look like? How to bring physical objects into the authoring process? What does a debugger or spectator module look like when editing reality? “Editing reality” is the overarching theme.  We need to think not only about how applications will have virtual and physical realities interacting, but also about the process of creating these applications.  

I leave you with a video of a mock application exploring how MR contextual apps could be created inside MR. By reusing some UI elements with which many are familiar, this is a little step towards pure virtual UIs. A frame around the working area acts as a container, and the UI is anchored outside so that dragging the frame towards a new physical object will bring the UI elements with it. The frame also guides the user and indicates where to look in order to activate/deactivate UI.  As mentioned in part II of this series, the idea is that UI should be clean, subtle, efficient and contextual.  This first step is an evolution, not a revolution so that it is useful and usable tomorrow, but we are continuously re-thinking UX and interaction in order to get to that future when physical screens will disappear (by “we” I mean our lead designer Greg. “I” the engineer am continuously figuring out how to implement it! 😄)

A mock application, rendered (or captured) with ARkit, showing how MR contextual apps could be authored in MR.

 

We are prototyping these and more ideas to understand the space and push for innovation and will keep you informed.

Article contributors: Greg Madison, Lead UX/IxD Designer and futurist; Colin Alleyne, Senior Technical Writer; and Sylvio Drouin, VP – Unity Labs.

Thanks to Unity Lab’s Authoring Tools Group who programmed the mock application shown in the video.

 

Comentários encerrados.

  1. I really enjoyed reading this series, for the in-depth insights and explanations, all those interesting links, etc. Thank you!
    Kindly,
    Chady

    1. Dioselin Gonzalez

      novembro 2, 2017 às 4:28 pm

      Glad you enjoyed it! Stay tuned for more

  2. what happend? just just ! i only see a rable and maney people. and h hand. this a arglass or future video!

  3. There is a big, hairy, fundamental “bug” underlying this vision: the requirement of presence.

    What do I mean? The big idea of The Web was time and location no longer matter. Within the context of a WIMP interface – and now iPhone – this enables tremendous reach, leverage, and amplification. Resources can be effortlessly marshaled at tremendous scale. Mobile interactions are unconscious and effortless. Everything is decoupled: sender and receiver of a message need not be present in time/space.

    MR/VR discards this leverage. This is its achilles heal. Time and space come roaring back demanding our constant undivided attention with significant cognitive load.

    To experience the vision you layout: I must be in Beijing. I must be in Paris. I must be in Brooklyn.

    Every action is one to one with an outcome. There is no amplification. Steve Jobs envisioned a computer as a “bicycle for the mind” as a lovely metaphor for this amplification. None exists in MR/VR.

    How do you intend to address this?

    Best,
    Doug Turner
    Elastic Image Software

    1. Dioselin Gonzalez

      outubro 11, 2017 às 6:15 pm

      Hi Douglas,

      Interesting thoughts! I see your point that the “next computing platform” should not bring back limitations that we have surpassed.

      First, re: time and (physical) space as a requirement. A couple of days ago we visited the pyramids of Egypt through Google Earth VR. With better photogrammetry, computer vision, machine learning, light field displays and haptic feedback, synthetic reality can –and will– become indistinguishable from physical reality. Additionally, as we mentioned in this article “cloud services will allow storing a persistent matrix, i.e. a digital copy of the physical world, and the synchronized shared mixed realities.” That matrix does not have to be a captured instant in time, instead we will be able to rewind and fast-forward it.

      So, if an AR experience in Beijing, Paris or Brooklyn happened 3 days ago, I enter the digital copy of the physical world to go there and then to start the AR experience. AR inside VR :) If the event is happening right now, people there see me as a hologram and we interact.

      Second, I want to mention we do not advocate for location-based AR to be the one and only way to exchange information. There are many scenarios in which this is not useful. As mentioned in Part II, we need to think ‘Human centric’ and only develop useful scenarios that add value, as opposed to gimmicks.

  4. Nice article, Mixed Reality has great future, Mixed Reality Market worth 453.4 Million USD by 2020

    Most of the large players in the reality market have invested in the mixed reality technology. For instance in January 2015, the Microsoft Corporation (U.S.) announced the launch of its mixed reality prototype product named HoloLens. The device features a see-through, holographic display and advanced sensors that map the physical environment. Moreover, other players such as Atheer, Inc. (U.S.), Meta Company (U.S.), Daqri LLC (U.S.), and Magic Leap, Inc. (U.S.) have developed prototypes to cater to various industrial and enterprise related needs.
    Download PDF Brochure: http://cutt.us/lrfAL

    This report provides a detailed analysis of the overall mixed reality market and its segmentation done on the basis of application and geography.

    The objective of the research study was to analyze the market trends for each of the industries, the growth rates of the various applications (industrial, aerospace & defense, medical, architecture, consumer, and others as well as the demand comparison of these industries.

    Apart from the market segmentation, the report also covers the market’s value chain with a detailed process flow diagram, and the market dynamics such as the drivers, restraints, and opportunities pertaining to the overall mixed reality market.