Table of Contents

Custom camera implementation in Unity —— External frame data source

Through the external frame data source (ExternalFrameSource), developers can extend EasyAR Sense with custom camera implementations to support specific head-mounted devices or other input devices. The following content introduces the type structure and interface definition of the external frame data source.

Before you begin

  • Understand the basic concepts of custom camera.
  • Understand the basic concepts, types, and runtime selection methods of frame source.

External frame data source types

---
  config:
    class:
      hideEmptyMembersBox: true
---
classDiagram
  class FrameSource {
    <<abstract>>
  }
  class ExternalFrameSource {
    <<abstract>>
  }
  class ExternalDeviceFrameSource  {
    <<abstract>>
  }
  class ExternalDeviceMotionFrameSource:::EasyAR {
    <<abstract>>
  }
  class ExternalDeviceRotationFrameSource:::EasyAR {
    <<abstract>>
  }
  class ExternalImageStreamFrameSource:::EasyAR {
    <<abstract>>
  }

  ExternalFrameSource --|> FrameSource
  ExternalDeviceFrameSource --|> ExternalFrameSource
  ExternalDeviceMotionFrameSource --|> ExternalDeviceFrameSource
  ExternalDeviceRotationFrameSource --|> ExternalDeviceFrameSource
  ExternalImageStreamFrameSource --|> ExternalFrameSource
  
  classDef EasyAR fill:#6e6ce6,stroke:#333,color:#fff

The above diagram shows the type structure of external frame data sources.

Based on the input data, external frame data sources can be divided into two main categories:

  • Image and device motion data input extension
    • Implemented by inheriting ExternalDeviceMotionFrameSource: The device and device SDK provide 6DoF motion tracking. The virtual camera's transform and other controls are handled by the device SDK.
    • Implemented by inheriting ExternalDeviceRotationFrameSource: The device and device SDK provide 3DoF rotation tracking. The virtual camera's transform and other controls are handled by the device SDK.
  • Image input extension
    • Implemented by inheriting ExternalImageStreamFrameSource: Only provides image input. The virtual camera's transform and other controls are handled by EasyAR.

When integrating these external frame data sources, the available AR features differ:

  • Image and device motion data input extension ExternalDeviceMotionFrameSource
    • Mega
    • Motion tracking (provided by the device itself)
    • Sparse spatial map
    • Dense spatial map
    • Image tracking (with motion fusion)
    • Image cloud recognition
    • Object tracking (with motion fusion)
  • Image and device motion data input extension ExternalDeviceRotationFrameSource
    • Mega
    • Image tracking (without motion fusion)
    • Image cloud recognition
    • Object tracking (without motion fusion)
  • Image input extension ExternalImageStreamFrameSource
    • Image tracking (without motion fusion)
    • Image cloud recognition
    • Object tracking (without motion fusion)

External frame data source interface definition

When creating an external frame data source, the relevant interfaces must be implemented. The following describes the definition and usage of these interfaces.

Device definition

  • FrameSource.IsHMD: Defines whether it is a head-mounted display
    Set to true if and only if the device is a head-mounted display.
    If the device is a head-mounted display, diagnostic information will be displayed on a 3D board in front of the camera instead of on the screen. Some AR features may behave slightly differently on head-mounted display devices.

  • FrameSource.Display: Defines the display system
    Provides information such as the rotation of the current display.
    You can use Display.DefaultSystemDisplay or Display.DefaultHMDDisplay to obtain default display information.
    Typically, Display.DefaultHMDDisplay can be used on head-mounted displays.

Availability

  • FrameSource.IsAvailable: Availability
    Used to determine whether the frame source is available.
    If a frame source is not available on the current running device or environment, this value should be false.
    If this value equals Optional<bool>.Empty, the FrameSource.CheckAvailability() coroutine will be called, and FrameSource.IsAvailable should be updated before the coroutine ends.
    The availability interface will be used during session assembly, and unavailable components will not be selected, and their methods will not be called during session runtime.
  • FrameSource.CheckAvailability() (optional): Coroutine to check if the frame source is available
    Called when FrameSource.IsAvailable equals Optional<bool>.Empty. The session assembly process will be blocked until this coroutine ends.

Session origin

Virtual camera

  • FrameSource.Camera: Virtual camera
    The camera is not controlled by the session. The camera's transform, projection matrix, and background image rendering should be controlled by external code.
    This camera is only used on head-mounted displays to display diagnostic text in front of the eyes.
    When ExternalDeviceFrameSource.OriginType is XROrigin, it does not need to be defined. EasyAR will automatically use the camera defined in the Unity XR framework.

Physical camera

  • FrameSource.DeviceCameras: Physical camera parameters
    The physical camera that provides camera frame data. If the camera frame data is provided by multiple cameras, the list needs to include all physical cameras.
    It is necessary to ensure that the correct physical camera parameters can be obtained when FrameSource.CameraFrameStarted is true.
  • FrameSource.CameraFrameStarted: Whether the camera frame has started input
    Returns true after the physical camera is ready and can input data to EasyAR, and returns false after the physical camera stops running. When FrameSource.CameraFrameStarted is false, EasyAR will not work. When FrameSource.CameraFrameStarted is true, it must be ensured that FrameSource.DeviceCameras data can be accessed and camera frame data is continuously input to EasyAR. If EasyAR detects that the camera frame has no input for a long time, a warning will pop up to help users determine the problem when the function is unresponsive.

The physical camera parameters need to be the same as the real device camera.

  • FrameSourceCamera.CameraType: Physical camera type
    Generally, for non-front cameras, such as on a headset, choose the rear camera.
  • FrameSourceCamera.CameraOrientation: The angle by which the physical camera image needs to be rotated clockwise to display in the natural orientation of the device
    The range is [0, 360).
  • FrameSourceCamera.FrameSize: Image size
  • FrameSourceCamera.FrameRateRange: Frame rate range
    Define x as the lower bound of the frame rate range and y as the upper bound of the frame rate range.
  • DeviceFrameSourceCamera.AxisSystem: The coordinate axis system used for head/physical camera pose and physical camera extrinsic parameters
    All matrices must use the same coordinate axis system. If the data definition used does not conform to a known system, a coordinate axis transformation is required before passing it to EasyAR.
  • DeviceFrameSourceCamera.Extrinsics: Physical camera extrinsic parameters
    Generally, it is a calibrated matrix. Its coordinate axis should conform to the definition of DeviceFrameSourceCamera.AxisSystem. If the coordinate axis definition of the extrinsic parameters is different from the actual pose's coordinate axis definition or they do not conform to the definition of DeviceFrameSourceCamera.AxisSystem, a coordinate axis transformation is required before setting this value.

Session start and stop

  • FrameSource.OnSessionStart(ARSession): Handle session start event
    Effective when this frame source is selected during session assembly.
    Can be used for delayed initialization, performing AR-specific initialization work in this method.
  • FrameSource.OnSessionStop(): Handle session stop event
    Effective when this frame source is selected during session assembly.
    Can be used to destroy resources created in FrameSource.OnSessionStart(ARSession) and during session operation, and restore internal state. This method is guaranteed to be called before session destruction. If the frame source is destroyed before the session, this method will not be called, and the session will enter the Broken state.

Input frame

  • ExternalFrameSource.TryAcquireBuffer(int): Try to acquire a memory block from the memory pool
    This memory block is usually used to store the image data of the camera frame and input it to EasyAR.
  • ExternalFrameSource.ReceivedFrameCount: The count of camera frames received by EasyAR
    EasyAR will use it to check the health of the device camera frame input. It can be used for debugging. If this value stops increasing, it usually means the device has stopped inputting data to EasyAR.

Unity messages

When using the following messages in scripts, ensure that the base class implementation is called:

Next steps