Cameras and input frame data sources in Unity — frame source (Frame Source)
The frame source is the provider of camera and input frame data in Unity. This article introduces the basic concepts, types, and selection methods of frame sources at runtime.
Before you start
- Understand the basic concepts, composition, and workflow of AR Session.
- Understand basic concepts such as cameras, input frame.
What is a frame source
The frame source (FrameSource) is the provider of input frames (InputFrame), abstracting cameras and other devices or functions that provide input frame data.
The diagram below shows the position of the frame source in the session:
flowchart LR
F[Frame Source]
A((Input Frame))
B[Session]
C([Camera])
O([Origin])
T([Target])
F --> A
A --> B
B -. transform .-> C
B -. transform .-> O
B -. transform .-> T
style F fill:#6e6ce6,stroke:#333,color:#fff
A frame source may only provide data for downstream AR features, or it may implement some AR features itself, such as motion tracking. Some frame sources provide control interfaces for camera devices, allowing users to select camera parameters, such as resolution, focus mode, etc.
Types of frame sources
Based on the Unity package that provides the frame source, frame sources can be divided into two categories:
- Built-in frame sources: Frame sources provided by the EasyAR Sense Unity plugin package, which typically support most common usage scenarios and some headsets.
- External frame sources: Frame sources provided by the EasyAR Sense Unity plugin extension package, typically used to support specific headset devices. Often, external frame sources are provided by headset manufacturers or third-party developers.
Different from external frame sources, custom cameras are not necessarily provided externally; some built-in frame sources are also custom cameras.
A frame source can provide motion data of different degrees of freedom: 0DoF, 3DoF, 5DoF, and 6DoF. The same frame source may provide motion data of different degrees of freedom in different working states.
The table below lists the frame sources provided by EasyAR:
| Name | Built-in | Custom camera | Motion data | Description |
|---|---|---|---|---|
| CameraDeviceFrameSource | Yes | No | None (0DoF) | Ordinary camera, supports front and rear cameras and PC |
| EditorCameraDeviceFrameSource | Yes | No | None (0DoF) | Ordinary camera, only supports debugging in the editor |
| FramePlayer | Yes | No | Determined by playback file | Plays back EIF files to simulate runtime |
| ThreeDofCameraDeviceFrameSource | Yes | No | 3DoF | Provides 3DoF tracking capability |
| InertialCameraDeviceFrameSource | Yes | No | 5DoF | Provides inertial navigation capability |
| MotionTrackerFrameSource | Yes | No | 6DoF | Provides motion tracking implemented by EasyAR |
| ARCoreFrameSource | Yes | No | 6DoF | Provides motion tracking from ARCore |
| ARKitFrameSource | Yes | No | 6DoF | Provides motion tracking from ARKit |
| AREngineFrameSource | Yes | Yes | 6DoF | Provides motion tracking from AR Engine |
| VisionOSARKitFrameSource | Yes | Yes | 6DoF | Provides motion tracking from VisionOS ARKit 1 |
| XREALFrameSource | Yes | Yes | 6DoF | Provides motion tracking for XREAL devices 1 |
| ARCoreARFoundationFrameSource | Yes | Yes | 6DoF | Provides motion tracking from ARFoundation corresponding to ARCore |
| ARKitARFoundationFrameSource | Yes | Yes | 6DoF | Provides motion tracking from ARFoundation corresponding to ARKit |
| PicoFrameSource | No | Yes | 6DoF | Provides motion tracking for Pico devices 1 |
| RokidFrameSource | No | Yes | 6DoF | Provides motion tracking for Rokid devices 1 |
Runtime frame source selection
The scene hierarchy of the session contains one or more frame source components. During session runtime, not all frame source components are used.
The screenshot below shows a scene hierarchy with a single frame source component:
The screenshot below shows a scene hierarchy containing multiple frame source components:
Each frame data source has different functions, which also determines their applicable usage scenarios and devices. During session assembly, one and only one of these components will be selected as the frame data source for the session.
The AssembleOptions.FrameSourceSelection property defines the selection method for the frame data source during session runtime:
| Name | Method |
|---|---|
| Auto (default) | Automatic selection, selects the first available and active child node in transform order. |
| Manual | Manual specification. Only child nodes of the session can be specified. |
| FramePlayer | Use FramePlayer. |
Tip
The transform order of Unity objects can be determined using Transform.GetSiblingIndex(), or by the order of objects in the Hierarchy view, but the following option must be turned off (it is off by default): Edit > Preferences > General > Enable Alphanumeric Sorting.
During session assembly, the frame data source is selected after going through the following steps:
- The session traverses its child nodes and collects all active frame data source components in transform order.
- Filter the candidate list according to the source selection strategy (AssembleOptions.FrameSource) in AssembleOptions:
- Auto (default): Retain all candidates.
- Manual: Only retain the manually specified frame data source.
- FramePlayer: Replace the candidate list with FramePlayer.
- Filter the candidate list again, removing the following components:
- Components that are disabled by the component itself.
- All custom camera components when custom camera is turned off (AssembleOptions.EnableCustomCamera is false).
- (Android platform) If the timeout setting of AssembleOptions.DeviceList is greater than 0, and the candidate list contains MotionTrackerFrameSource, ARCoreFrameSource, or AREngineFrameSource, it will attempt to download the latest device support list. After the download is updated, the availability of these frame data sources may change. After the download completes or times out, proceed to the next steps.
- Check the availability of the remaining candidate components in order (by calling FrameSource.CheckAvailability() and accessing FrameSource.IsAvailable).
- Select the first frame data source that passes the availability check.
The disabling conditions of the component itself are defined internally by the component. Common situations include:
- Running in an unsupported system, for example, AREngineFrameSource will be disabled in non-Android systems.
- The necessary third-party SDK is not installed, for example, XREALFrameSource will be disabled when the XREAL SDK is not installed.
- The configured conditions are not met, for example, MotionTrackerFrameSource will be disabled when the device's MotionTrackerCameraDeviceQualityLevel is lower than MotionTrackerFrameSource.DeviceQualityLevel.
If no frame data source is ultimately selected, the session will enter the Broken state, and the value of the BrokenReason field in the session report will be NoAvailabileFrameSource.
Note
After the device list update is completed, if the device list changes, the availability of frame data sources may also change. You can refer to Device Support and Session Report to understand the session's behavior at this time.
Next steps
- Try adding a group of frame sources in the scene
Related topics
- Device support and session report
- EasyAR headset support
- Create an external frame source to use a custom camera
-
For device support, refer to EasyAR headset support.↩↩↩↩