Context is everything: Cross-domain data mapping for augmented reality

AI is enjoying a heyday, although the path has yet to be paved in terms of how it can be leveraged for immersive experiences. AI is not the solution to every problem, but it does provide the missing link needed to create next-generation augmented experiences.

Considering this AI boom, I’d like to talk about the concept of “context” as it relates to creating augmented reality (AR) experiences. A context is a collection of parameters describing a domain that can be encoded into machine-processable data. A domain can be literally anything, but in the AR space, three critical domains exist: physical, virtual, and human.

The physical domain is the world we live in. The context for a specific location may include world coordinates (that is, a geographical position), environment scan data, object positions, weather, or images of the surroundings – whatever real-world parameters are relevant to support the generation of a solution to a specific need.

The virtual domain contains any data that has a useful correlation with a location in the real world. This is a broad definition, but that’s the idea: AR experiences don’t need complex 3D assets or models to provide value. Any kind of location metadata can be the basis for an experience, for example, rainfall data or the location of stock in a store.

Finally, the human domain is the body of human requirements, expressed in terms a machine can understand. This is the jumping-off point for AI, where natural language processing (NLP) and generative pre-trained transformer (GPT) models play a key role in converting the human context into machine language. The human domain also encompasses how machine-generated data is communicated.

Generating a domain context is a relatively straightforward task. Where it gets tricky is ensuring the relationships between components are usable: physical and virtual coordinate systems must align, digital twins must be up to date with the physical world, human descriptions must be mappable to trainable behaviors, etc.

Both software and hardware related to emerging technologies, including AI, robotics, and the Internet of Things, are evolving rapidly. Until standards (for interoperability, for example) and best practices are in place governing their implementation, effective usage and compatibility relies on the skilled design of networked components. But once this system is designed, you have the generalized foundation for creating augmented reality experiences for any application, be it industry, retail, or general productivity improvement.

An example of how AI enables cross-domain mapping would be an individual pointing at an object in the distance. This is a physical context that many technologies can provide, but the gesture in itself doesn’t have intrinsic meaning and is not sufficient for defining the problem that needs solving. It could be in reference to a direction of travel, or an inquiry about an object. The context when correlated with language like, “How do I get there?” now forms a complete query. Thus, AI can process the physical data guided by human context data to “understand” the natural interactions we all perform daily without thinking about them and generate an appropriate response. That transparency of request/response is elevating all forms of AR experience, with the ultimate goal of making our lives easier.

Let’s explore some examples of how context can be defined for different applications. AI’s primary role is in the human domain, processing user requests, anticipating user needs, drawing on relevant data, and facilitating communication between people and devices.

Room painting: A user wants to paint a space and would like to know the quantity of materials needed. Specifics: They have a device that can measure the space and issue a voice command asking how much paint is needed.

• Physical: Lidar scan of physical space
• Virtual: Digital twin of the space created on the fly based on physical scan that includes windows, doors, and walls to accurately determine the wall surface area
• Human: The correlation between square footage and how much surface area a canful of paint can cover

Fitness routing: A user requests a run variation from the route usually taken. Specifics: The user has a headset that’s able to determine the user’s location, has a record of previous routes, and can project visual information.

• Physical: User location, and recordings of previous runs
• Virtual: Maps of the area that provide information on trails and sidewalks
• Human: Understanding of what constitutes a route to allow calculation of a new one

Airport optimization: Situational awareness and automation to improve operations management. A user needs just-in-time prompts for conducting airfield activities safely. Specifics: The user has a wrist wearable that can determine the user’s location, and has a data connection to a central operational digital twin.

• Physical: Locations of users, aircraft, assets, and physical world objects
• Virtual: Digital twin of airport enabling simulation prediction, navigation, locations of points of interest, and geospatial processing
• Human: Understanding of the mission, challenges, and key safety objectives

As one can see in these examples, the value of real-time 3D extends well beyond the generation of impressive visuals. It’s the core engine for processing cross-domain contexts to generate spatial solutions to problems. Given that we live in a 3D world, it’s not surprising that real-time 3D plays a central role.

Unity as a core data engine has massive traction in the gaming market, and so its applicability to non-game use cases is often overlooked. As wearables, devices, and AI models advance technologically, capturing more and better data, ever richer contexts will be defined, generating more precise solutions. Unity will be the primary tool for collecting this data for creating the experiences that make our lives better at work and play.

We’re excited by what our developers will create, and hopefully this blog post provided some ideas on how to structure your next-generation experiences.

For more inspiration on how AI can drive immersive experiences, check out the potential of digital twins. And don’t forget to sign up for Unity AI Beta Program.