Table of Contents

EasyAR Unity head-mounted display extension package

This document introduces the concepts, scope of capabilities, and prerequisite knowledge required to create head-mounted display extension packages for EasyAR Unity.

Before you begin

What is the EasyAR Unity headset extension package

The EasyAR Unity headset extension package is a Unity package containing a series of code and samples to help you utilize the features of EasyAR Sense on your headset device. Through this extension package, you can integrate most features of EasyAR Sense (such as image tracking, dense spatial map, etc.) into your device, thereby leveraging the powerful AR capabilities provided by EasyAR.

Using the EasyAR Unity headset extension package is one of the ways EasyAR supports headsets. The diagram below illustrates the overall architecture of EasyAR in Unity and the position of the headset extension package within it.

block
  columns 4
  block:groupApp:4
    block:groupAppWrapper
      space
      App1["EasyAR + Device A<br>App"]
      space
      App2["EasyAR<br>App"]
      space
      App3["EasyAR + Device B<br>App"]
    end
  end
  
  block:groupSensePluginExtension
    columns 1
    SensePluginExtension["EasyAR Sense Unity Plugin<br>Extension for Device A"]
    space
  end
  block:groupSensePlugin
    columns 1
    SensePlugin["EasyAR Sense Unity Plugin"]
    space
  end
  block:groupXRI
    columns 1
    XRI["XR Interaction Toolkit"]
    space
  end
  block:groupARF
    columns 1
    ARF["AR Foundation"]
    space
  end
  
  block:groupDeviceAUnity
    columns 1
    DeviceAUnity["Device A<br>Unity SDK"]
    space
  end
  block:groupSense
    columns 1
    Sense["EasyAR Sense"]
    block:groupSenseWrapper
      MDeviceB["Device B<br>CameraDevice"]
      Others["..."]
    end
  end
  block:groupXRSubsystem:2
    columns 1
    XRSubsystem["XR Subsystems"]
    XRSDK["Unity XR SDK"]
  end

  block:groupSystem:4
    columns 1
    System["Native Library"]
    block:groupSystemWrapper
      space
      DeviceA["Device A<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Library&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"]
      space
      space
      DeviceB["Device B<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Library&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"]
      space
    end
  end
  
  SensePluginExtension --> App1
  SensePlugin --> App1
  SensePlugin --> App2
  SensePlugin --> App3
  ARF --> App3
  XRI --> App1
  XRI --> App3
  groupSense --> SensePlugin
  groupDeviceAUnity --> SensePluginExtension
  SensePlugin --> SensePluginExtension
  DeviceA --> groupDeviceAUnity
  DeviceA --> XRSDK
  XRSubsystem --> ARF
  XRSubsystem --> XRI
  DeviceB --> MDeviceB
  DeviceB --> XRSDK
  
  style groupApp fill:none,stroke:none,stroke-width:0px
  style groupAppWrapper fill:none,stroke:none,stroke-width:0px
  style groupSensePlugin fill:none,stroke:none,stroke-width:0px
  style groupARF fill:none,stroke:none,stroke-width:0px
  style groupXRI fill:none,stroke:none,stroke-width:0px
  style DeviceAUnity fill:none,stroke:none,stroke-width:0px
  style Sense fill:none,stroke:none,stroke-width:0px,color
  style groupSenseWrapper fill:none,stroke:none,stroke-width:0px
  style XRSubsystem fill:none,stroke:none,stroke-width:0px
  style System fill:none,stroke:none,stroke-width:0px
  style groupSystemWrapper fill:none,stroke:none,stroke-width:0px
  style groupSensePluginExtension fill:none,stroke:none,stroke-width:0px  

  classDef EasyAR fill:#6e6ce6,stroke:#333,color:#fff
  class SensePluginExtension EasyAR

The diagram lists two typical headset support approaches: connecting to the device SDK through the Unity headset extension package (Device A), and directly connecting to the device SDK within the EasyAR Sense library (Device B). This document primarily focuses on the former.

Can I create my own headset extension package?

Currently, the AR/VR/MR/XR industry lacks a highly unified interface solution. While OpenXR is a strong candidate, the evolution of the specification and industry implementation still requires time. Therefore, it's generally not straightforward to run EasyAR directly on commercially available devices; there's a high likelihood of missing data interfaces. As the industry develops, some emerging devices might offer good interface support. For instance, in 2024, Apple opened up relevant interfaces for the Vision Pro, which are sufficient to support EasyAR operation, though using them still requires some specialized knowledge.

If you are unable to make a judgment, it is recommended to contact the hardware manufacturer or EasyAR business to obtain corresponding device support.

If you are a hardware manufacturer and wish to support EasyAR functionality on your device, you can refer to the following documentation to create a headset extension package. This will enable most EasyAR features to run on your device. While providing data and interface specifications, this document does not dictate all implementation details. Any implementation approach or interface definition is open for discussion; welcome to contact us via business channels.

The hardware covered by this document must inherently possess motion tracking or SLAM capabilities. EasyAR's functionality relies on running atop robust device tracking capabilities. It is generally not advisable to rely on EasyAR features to optimize device tracking, as this creates a circular dependency that can theoretically amplify errors and lead to overall system instability. If the device itself lacks motion tracking capabilities, the support solution falls outside the scope of this document. Please contact us via business channels if needed.

Capability boundaries of the headset extension package

The goal of the headset extension package is to enable most features of EasyAR Sense to run on your device. To achieve this goal, you need to understand the capability boundaries of the headset extension package.

Head-mounted display extension package contents

You will be implementing extensions including:

  • A series of code that uses custom camera functionality to capture data from your device API and feed it into EasyAR Sense.
  • In Unity, the head-mounted display extension utilizes external frame data source and a data flow defined by the EasyAR Sense Unity Plugin to simplify custom camera development.
  • In Unity, the head-mounted display extension is a Unity package containing runtime scripts, editor scripts, and extended samples. You or EasyAR can distribute it to downstream users.
Tip

If you prefer not to expose integration details to external systems, you may contact EasyAR for discussion. Direct integration using the C interface within EasyAR Sense is feasible and has precedents.

During the implementation of the extension, you might:

  • Modify the interface design and internal implementation of your SDK.
  • Discuss and confirm data acquisition and usage plans with your team.
  • Spend a significant amount of time validating data correctness rather than writing code.

After completing the extension, you will observe:

  • Most EasyAR Sense functionalities work on your device, leveraging its motion tracking capabilities.
  • EasyAR cloud services supported within EasyAR Sense are functional on your device.
  • Only EasyAR XR licenses are usable. Personal, Professional, and Classic licenses will not function on your device.
  • All EasyAR license limitations applicable when using a custom camera apply similarly to your device.

What the headset extension does not include

This extension cannot function without EasyAR Sense:

  • The headset extension does not operate independently; as a dependency, EasyAR Sense is also required. In Unity, the EasyAR Sense Unity Plugin must be used.
  • It does not directly call EasyAR cloud-based service APIs (such as the EasyAR Mega localization service); these calls are handled internally within EasyAR Sense.
  • In Unity, it does not directly call interface methods for AR functionalities (like image tracking); these are handled internally within the EasyAR Sense Unity Plugin.
  • In Unity, it does not modify the transform of scene objects or tracked targets; these are handled internally within the EasyAR Sense Unity Plugin.

This extension cannot function without your device SDK:

  • In Unity, neither the headset extension nor the EasyAR Sense Unity Plugin modifies the transform of the scene camera; this must be done within your device SDK or its dependency chain.

Certain EasyAR functionalities remain unavailable through the headset extension:

  • Surface tracking functionality will not be available.
  • EasyAR's own motion tracking will not be available.
  • Plane detection (part of EasyAR motion tracking) will not be available.

How to use Mega on my device?

Running Mega on devices is a concern for many users. In Unity, the Mega service is a functional module built on top of various foundational capabilities of EasyAR Sense. Therefore, as long as your device fully supports EasyAR Sense, Mega will also be supported.

Generally, it is not recommended to initially verify device support for Mega by directly running the Mega sample on the device. This is because Mega comprehensively utilizes all input data and has a relatively high tolerance for errors in this data. Directly running the Mega sample could lead to unreasonable performance due to mismatched data interfaces or poor data quality. This would make it difficult to pinpoint the source of the problem, creating challenges for future debugging.

Important

The Mega service has certain requirements for the device's motion tracking capability. If the device's motion tracking capability is poor, Mega's performance will also be affected. In large-scale AR scenarios, special attention should be paid to performance differences between indoor and outdoor environments.

Important

Mega typically serves large-scale spatial scenarios. Therefore, extra attention must be paid to the display effect of long-distance objects and objects when turning the head or moving. If the device's display system has significant errors, users will perceive that virtual objects cannot correctly align with real objects, even if Mega itself is functioning normally.

Required background knowledge and team configuration

Creating a head-mounted display (HMD) extension package is not a simple task; it requires you and your team to engage in in-depth work across multiple domains. Typically, completing an HMD extension requires the involvement of Unity developers alongside team members outside of Unity development. Due to the lack of standards, modifications solely on the 3D engine are often insufficient to complete an HMD extension. It is recommended to involve low-level development engineers, such as system engineers and SDK engineers, from day one.

Building AR/VR devices requires domain-specific knowledge. Similarly, running and validating EasyAR Sense on a device will require you or your team to be experts in the following areas:

  • Your device's physical structure and rendering system
  • Camera system geometry
  • SDK development
  • General Android debugging skills, such as adb (Mainland China, International)

If you are working on Unity, you will also need to know about:

Additionally, knowledge in these areas will help you better understand the system, especially how to send the correct data to EasyAR:

Next steps

In the following articles, you will learn the complete process for creating a headset extension package: