Table of Contents

Custom camera implementation in Unity —— External frame data source

Through the external frame data source (ExternalFrameSource), developers can extend EasyAR Sense with a custom camera implementation to support specific head-mounted devices or other input devices. The following describes the type structure and interface definitions of the external frame data source.

Before you begin

  • Understand the basic concepts of custom camera.
  • Understand the basic concepts, types, and selection method at runtime of frame source.

External frame data source types

---
  config:
    class:
      hideEmptyMembersBox: true
---
classDiagram
  class FrameSource {
    <<abstract>>
  }
  class ExternalFrameSource {
    <<abstract>>
  }
  class ExternalDeviceFrameSource  {
    <<abstract>>
  }
  class ExternalDeviceMotionFrameSource:::EasyAR {
    <<abstract>>
  }
  class ExternalDeviceRotationFrameSource:::EasyAR {
    <<abstract>>
  }
  class ExternalImageStreamFrameSource:::EasyAR {
    <<abstract>>
  }

  ExternalFrameSource --|> FrameSource
  ExternalDeviceFrameSource --|> ExternalFrameSource
  ExternalDeviceMotionFrameSource --|> ExternalDeviceFrameSource
  ExternalDeviceRotationFrameSource --|> ExternalDeviceFrameSource
  ExternalImageStreamFrameSource --|> ExternalFrameSource
  
  classDef EasyAR fill:#6e6ce6,stroke:#333,color:#fff

The above diagram shows the type structure of external frame data sources.

Based on different input data, external frame data sources can be divided into two main categories:

  • Image and device motion data input extension
    • Implemented by inheriting ExternalDeviceMotionFrameSource: The device and device SDK provide 6DoF motion tracking. The virtual camera's transform and other controls are handled by the device SDK.
    • Implemented by inheriting ExternalDeviceRotationFrameSource: The device and device SDK provide 3DoF rotation tracking. The virtual camera's transform and other controls are handled by the device SDK.
  • Image input extension
    • Implemented by inheriting ExternalImageStreamFrameSource: Only provides image input. The virtual camera's transform and other controls are handled by EasyAR.

When integrating these external frame data sources, the available AR features differ:

  • Image and device motion data input extension ExternalDeviceMotionFrameSource
    • Mega
    • Motion tracking (provided by the device itself)
    • Sparse spatial map
    • Dense spatial map
    • Image tracking (with motion fusion)
    • Cloud image recognition
    • Object tracking (with motion fusion)
  • Image and device motion data input extension ExternalDeviceRotationFrameSource
    • Mega
    • Image tracking (without motion fusion)
    • Cloud image recognition
    • Object tracking (without motion fusion)
  • Image input extension ExternalImageStreamFrameSource
    • Image tracking (without motion fusion)
    • Cloud image recognition
    • Object tracking (without motion fusion)

External frame data source interface definition

When creating an external frame data source, relevant interfaces must be implemented. The definitions of these interfaces and how to use them are described below.

Device definition

  • FrameSource.IsHMD: Defines whether it is a head-mounted display
    Set to true only on head-mounted display devices.
    If the device is a head-mounted display, diagnostic information will be displayed on a 3D board in front of the camera instead of on the screen. Some AR features may operate slightly differently on head-mounted display devices.

  • FrameSource.Display: Defines the display system
    Provides information such as the current display's rotation.
    Use Display.DefaultSystemDisplay or Display.DefaultHMDDisplay to obtain default display information.
    Typically, Display.DefaultHMDDisplay can be used on head-mounted displays.

Availability

  • FrameSource.IsAvailable: Availability
    Used to determine if the frame source is usable.
    If a frame source is unavailable on the current running device or environment, this value should be false.
    If this value equals Optional<bool>.Empty, the FrameSource.CheckAvailability() coroutine will be called. FrameSource.IsAvailable should be updated before the coroutine ends.
    The availability interface is used during session assembly. Unavailable components will not be selected, and their methods will not be called during session runtime.
  • FrameSource.CheckAvailability() (Optional): Coroutine to check the availability of the frame source
    Called when FrameSource.IsAvailable equals Optional<bool>.Empty. The session assembly process will be blocked until this coroutine ends.

Session origin

Virtual Camera

  • FrameSource.Camera: Virtual Camera
    The camera is not controlled by the session. The camera's transform and projection matrix, as well as image background rendering, should be controlled by external code.
    This camera is only used on head-mounted displays to display diagnostic text in front of the eyes.
    When ExternalDeviceFrameSource.OriginType is XROrigin, definition is not required. EasyAR will automatically use the camera defined in Unity XR framework.

Physical camera

  • FrameSource.DeviceCameras: Physical camera parameters
    Physical cameras providing camera frame data. If camera frame data is provided by multiple cameras, the list must include all physical cameras.
    Ensure correct physical camera parameters can be obtained when FrameSource.CameraFrameStarted is true.
  • FrameSource.CameraFrameStarted: Whether camera frame input has started
    Returns true after the physical camera is ready and can feed data to EasyAR, returns false after the physical camera stops. When FrameSource.CameraFrameStarted is false, EasyAR will not work. When FrameSource.CameraFrameStarted is true, ensure FrameSource.DeviceCameras data is accessible and continuously feeds camera frame data to EasyAR. If EasyAR detects no camera frame input for a prolonged period, it will issue a warning to help users decouple issues when functionality is unresponsive.

Physical camera parameters must match the real device camera.

Session startup and shutdown

  • FrameSource.OnSessionStart(ARSession): Handle session startup event
    Effective when this frame source is selected during session assembly.
    Can be used for lazy initialization; perform AR-specific initialization work in this method.
  • FrameSource.OnSessionStop(): Handle session shutdown event
    Effective when this frame source is selected during session assembly.
    Can destroy resources created in FrameSource.OnSessionStart(ARSession) and during session operation, and restore internal state. This method is guaranteed to be called before session destruction. If the frame source is destroyed before the session, this method won't be called, and the session will enter the Broken state.

Input frame

  • ExternalFrameSource.TryAcquireBuffer(int): Attempt to acquire a memory block from the memory pool
    This memory block is typically used to store image data for camera frames and pass it to EasyAR.
  • ExternalFrameSource.ReceivedFrameCount: Number of camera frames received by EasyAR
    EasyAR uses this to monitor the health of device camera frame input. Can be used for debugging; if this value stops increasing, it usually indicates that the device has stopped feeding data to EasyAR.

Unity messages

When using the following messages in scripts, ensure that base class implementations are called:

Next steps