Table of Contents

Camera and input frame data sources in Unity —— Frame source

The frame source is the provider of camera and input frame data in Unity. This article introduces the basic concepts, types, and runtime selection methods of frame sources.

Before you begin

What is a frame source

A frame source (FrameSource) is a provider of input frames (InputFrame), abstracting cameras and other devices or functionalities that provide input frame data.

The following diagram illustrates the position of a frame source in a session:

flowchart LR  
  F[Frame Source]  
  A((Input Frame))  
  B[Session]  
  C([Camera])  
  O([Origin])  
  T([Target])  
  F --> A  
  A --> B  
  B -. transform .-> C  
  B -. transform .-> O  
  B -. transform .-> T  

  style F fill:#6e6ce6,stroke:#333,color:#fff

A frame source may simply provide data for downstream AR functionalities to use, or it may itself implement some AR functionalities, such as motion tracking. Some frame sources provide control interfaces for camera devices, allowing users to select camera parameters, such as resolution, focus mode, etc.

Types of frame data sources

Based on the Unity package that provides the frame data source, frame data sources can be divided into two main categories:

  • Built-in frame data sources: Provided by the EasyAR Sense Unity plugin package, usually supporting most common usage scenarios and some headsets.
  • External frame data sources: Provided by the EasyAR Sense Unity plugin extension package, usually used to support specific headset devices. Often, external frame data sources are provided by headset manufacturers or third-party developers.

Different from external frame data sources, custom camera is not necessarily provided externally, as some built-in frame data sources also include custom cameras.

Frame data sources can provide motion data with different degrees of freedom: 0DoF, 3DoF, 5DoF, and 6DoF. The same frame data source may provide motion data with different degrees of freedom under different working conditions.

The table below lists the frame data sources provided by EasyAR:

Name Built-in Custom camera Motion data Description
CameraDeviceFrameSource Yes No None (0DoF) Regular camera, supports front/rear cameras and PC
EditorCameraDeviceFrameSource Yes No None (0DoF) Regular camera, only for debugging in the editor
FramePlayer Yes No Determined by playback file Plays back EIF files for simulation
ThreeDofCameraDeviceFrameSource Yes No 3DoF Provides 3DoF tracking capability
InertialCameraDeviceFrameSource Yes No 5DoF Provides inertial navigation capability
MotionTrackerFrameSource Yes No 6DoF Provides motion tracking implemented by EasyAR
ARCoreFrameSource Yes No 6DoF Provides motion tracking from ARCore
ARKitFrameSource Yes No 6DoF Provides motion tracking from ARKit
AREngineFrameSource Yes Yes 6DoF Provides motion tracking from AR Engine
VisionOSARKitFrameSource Yes Yes 6DoF Provides motion tracking from VisionOS ARKit 1
XREALFrameSource Yes Yes 6DoF Provides motion tracking for XREAL devices 1
ARCoreARFoundationFrameSource Yes Yes 6DoF Provides motion tracking from ARFoundation for ARCore
ARKitARFoundationFrameSource Yes Yes 6DoF Provides motion tracking from ARFoundation for ARKit
PicoFrameSource No Yes 6DoF Provides motion tracking for Pico devices 1
RokidFrameSource No Yes 6DoF Provides motion tracking for Rokid devices 1

Runtime frame data source selection

The session's scene hierarchy contains one or more frame data source components. During session runtime, not all frame data source components will be used.

The following screenshot shows a scene hierarchy with only a single frame data source component:

alt text

The following screenshot shows a scene hierarchy containing multiple frame data source components:

alt text

Each frame data source has different functionalities, which also determine their applicable usage scenarios and devices. During session assembly, one and only one of these components will be selected as the frame data source for the session.

The AssembleOptions.FrameSourceSelection property defines the method for selecting the frame data source during session runtime:

Name Method
Auto (default) Automatic selection, choosing the first available and active child node in transform order.
Manual Manual specification. Only child nodes of the session can be specified.
FramePlayer Use FramePlayer.
Tip

The transform order of Unity objects can be determined using Transform.GetSiblingIndex() or by observing the sorting of objects in the Hierarchy view, but the following option must be disabled (it is disabled by default): Edit > Preferences > General > Enable Alphanumeric Sorting.

During session assembly, the frame data source is selected after the following steps:

  1. The session traverses its child nodes and collects all active frame data source components in transform order.
  2. The candidate list is filtered based on the source selection strategy (AssembleOptions.FrameSource) in AssembleOptions:
    • Auto (default): All candidates are retained.
    • Manual: Only the manually specified frame data source is retained.
    • FramePlayer: Replace the candidate list with FramePlayer.
  3. The candidate list is further filtered by removing the following components:
  4. (Android platform) If the timeout setting of AssembleOptions.DeviceList is greater than 0, and the candidate list includes MotionTrackerFrameSource, ARCoreFrameSource, or AREngineFrameSource, the latest device support list will be attempted to download. After the download is updated, the availability of these frame data sources may change. After the download is completed or times out, proceed to the next steps.
  5. The remaining candidate components are checked for availability in list order (by calling FrameSource.CheckAvailability() and accessing FrameSource.IsAvailable).
  6. The first frame data source with an available check result is selected.

Among these, the conditions for a component to disable itself are defined internally by the component. Common scenarios include:

If no frame data source is ultimately selected, the session will enter the Broken state, and the BrokenReason field in the session report will have the value NoAvailabileFrameSource.

Note

After the device list update is completed, if the device list changes, the availability of frame data sources may also change. Refer to Device support and session report to understand the session's behavior in this case.

Next steps

  1. For device support, refer to EasyAR headset support.↩↩↩↩