How the Query system in Unity MARS does procedural layout
The Query system in Unity MARS simplifies the creation of augmented reality (AR) experiences that adapt to the real world around them. Based on constraints that you specify, it finds a way to lay out the scene in a given environment. Read on to learn more about how it works.
We designed Unity MARS to help creators make context-adaptable AR. We hoped to solve the primary issues AR developers face: defining content relative to the real world, iteration time and testing in environments, and adapting the way virtual content is laid out in varying real-world environments. We call this the adaptable content problem.
Queries are Unity MARS’ way of solving the adaptable content problem. For each distinct piece (or group of pieces) of virtual content that needs to integrate into a place in the real world, Unity MARS devises a query – a request for semantically tagged data that describes part of the real world. Queries have two high-level pieces:
- One or more conditions that define what data can match the query
- One or more actions that specify what to do when Unity MARS finds a match
Each entry in the Unity MARS Database represents a single part of the real-world environment around the user. These parts can take many forms – some of the most common are surfaces, faces, and image markers.
Each one of these entries is identified by a number called a data ID. They also can have any number of traits – named properties of a given data type. This model is conceptually similar to an entity component system – each trait is like a component.
Each proxy has a property called Exclusivity, which determines how it will use the data it matches against, and limits how the query handles trying to match against data that is already in use. Keeping track of data that’s already used allows the user to make sure different pieces of content go to different pieces of the real world. There are three exclusivity settings:
- Reserved is asking for exclusive access to this data – it needs to not already be in use by a Shared or Reserved proxy, and after it matches, it can only be used by other proxies in a read-only manner.
- Shared is asking to use this data, but not exclusively – other Shared proxies can already be using it, and use it after it matches.
- Read Only can always access any data, since it declares that it will not “use up” the option.
After a query matches, all the proxies within it are marked as “using” that piece of data in the manner their exclusivity specifies.
Every condition takes into account a single trait of the data it’s considering, and determines whether or not it fits the criteria. One of the primary benefits of authoring a Condition against a trait, such as position or size, instead of directly against trackable data like surfaces or faces, is that they are re-usable for any data that has that same trait.
Consider the Plane Size condition as an example.
It requires the trait “bounds2d”, which describes the acceptable size of the 2-dimensional oriented bounding box of the data segment the proxy is looking for, and has the data type Vector2 – two floating point numbers. This condition can be evaluated against any data is tagged as a «plane» and has the “bounds2d” trait, which is also present for image markers.
To begin, Unity MARS considers the available options that have the bounds2d trait, ruling out any that don’t fit within the minimum and maximum size.
Any options that fit within the specified bounds get assigned a rating – a number on a scale from 0 to 1 that describes how well they match, where 1 is perfect and 0 is not a match. Think of this as “closeness of fit.” For the Plane Size condition, bounds values that are closer to the middle of the range receive a higher rating than ones closer to the edges.
If there are multiple conditions on a proxy, each of them must be satisfied by the same data for the query to match. In addition, the ratings for each condition are combined to form the rating describing how well the proxy matches against a given data option.
For example, if you were to add an Alignment condition specifying that the surface the proxy matches against must be both horizontal and facing upwards, then the matched surface must satisfy the previous Plane Size condition as well as the alignment condition.
The system is always trying to give you the available match to your query that has the best rating when all conditions are taken into account together.
Groups and Relations
Proxies that are contained within a proxy group must all match at the same time, to different pieces of data. Additionally, you can define relations between any two proxies in the group.
As an example, the Elevation relation defines an acceptable range of difference in real-world elevation – in Unity, elevation is distance on the world’s Y axis. Here, we have an elevation relation between a proxy looking for a large plane, and another proxy above looking for a smaller plane define the elevation relation as the distance between them.
A relation functions much like a condition does, except that it looks at the data for both members of the relation.
The elevation relation looks at the “pose” trait to find the position in space of each option, determines whether they are within the acceptable range, and assigns a rating from 0 to 1 for each possibility that is considered. Just like with the Plane Size condition, pairs of data that are closer to the middle of the range are considered more ideal.
To determine how well each set of possibilities matches the overall query formed by all of the conditions and relations in a group, the scores for all relations in a group are combined, just as for conditions on a proxy, to average the score for each group.
Actions specify what Unity MARS will do when it finds a match. Like conditions, actions depend on the trait data they match against, but unlike conditions, they may specify multiple required traits.
The simplest action, which every proxy has by default, is the Set Pose action. It depends on the pose trait, of type Pose – a position and rotation in world space. As with conditions, this rules out any data options that do not have that trait.
Once the query matches, the Set Pose action receives the pose value for the match and moves the virtual content to that position. This is how matched queries localize themselves in the environment, as seen in this simulation.
Priority and resolving conflicts
You can establish some pieces of content as more important than others. Specifically, you can assign each proxy or proxy group a Priority setting. There are five priority levels, ranging from Lowest to Highest, with the default being Normal.
Priority is used as the first factor in determining which object should have first access to the data it matches best against, during the “conflict resolution” stage. If multiple proxies request the same data ID as their best match, Unity MARS looks at which one has the highest priority and assigns it to that one. If their priorities are equal, the data ID is assigned to the proxy that got the better match rating.
The problem can get fairly complex, even for a single proxy group. One helpful way to conceptualize what’s going on with the data is through graphs.
Here’s an example of a group that has both elevation and distance relationships. Each vertex represents a proxy, and each edge represents a relation between two proxies.
The problem of matching a group can be expressed in technical terms as a subgraph isomorphism search. In this case, the system is not just looking for any match for the subgraph, but for the best (highest-rated) instance of it in the larger graph below, which is the “search space.” This larger graph is derived from the set of valid instances of the relations being sought.
The top is with all layers combined, and below shows the layer defined by each relation split out.
Each vertex represents a data entity, and each edge represents a valid instance of one of the distance or elevation relations between two data entities.
More relations or more members of the group will result in more layers in the subgraph that is looking for a match, which are more complex to visualize.
Dealing with complexity
The problem of finding the most mathematically perfect match for a proxy group leads to combinatorial explosion as the number of group members and the size of the data set grow. This applies to a lesser degree to large scenes of proxies that are not grouped, since the entire scene is considered at once.
To ensure that the system continues to work with complex queries without hitching the application, the system does three things.
First, the work for solving queries is broken into named stages, which each do a small portion of the work, spread out over time.
Second, the system applies an approximation strategy as the complexity grows, meaning that the answer returned for a very complex group query will be the best that could be found within a reasonable amount of computation.
Third, the system solves the most complex and highest priority ones first, to maximize the likelihood of a good match for them. More complex queries can have more effects on the possible matches of the other queries that are considered in the same evaluation as them.
Learn more about the Query system in Unity MARS
Get more information on Unity MARS and the benefits it brings to AR developers. We also invite you to join us in the Unity Forums for further discussion related to the query system and procedural layout in Unity MARS.
You can get started with Unity MARS today with a 45-day free trial.