Table of Contents

The entry point to Unity AR —— AR Session

The AR session serves as the gateway to all AR functionalities. The following content will help you understand the fundamental concepts, composition, and operational flow of an AR Session. You will also learn how it relates to the AR Session in Unity AR Foundation and discover how the data flow of EasyAR Sense actually works within Unity.

What is an ar session

All AR processes (such as object tracking) are executed within the native library, specifically EasyAR Sense. The session is the primary entry point for AR functionality in Unity. It manages the operation and state of the AR system, including reading data from physical cameras and sensors, analyzing the real world, driving the movement and rendering of virtual cameras and other objects in the scene, among other tasks.

flowchart LR
  A((Image<br>and other data))
  B[Session]
  C([Camera])
  O([Origin])
  T([Target])
  A --> B
  B -. transform .-> C
  B -. transform .-> O
  B -. transform .-> T

[Optional] EasyAR session vs AR Foundation session

EasyAR session is the core component for using EasyAR in Unity, which can operate independently of any third-party or system AR features. In contrast, AR Foundation session is part of Unity’s XR framework and relies solely on functionalities provided by Unity XR plugins (such as ARKit or ARCore).

flowchart TD
  A1[EasyAR<br>AR Session]
  A2[EasyAR Sense]
  A1 --> A2

  B1[AR Foundation<br>AR Session]
  B2[ARKit Plugin]
  B3[ARCore Plugin]
  B1 --> B2
  B1 --> B3

When using EasyAR, it typically isn’t necessary to install or use AR Foundation simultaneously. Features like image tracking, motion tracking, etc., are independently provided by EasyAR Sense.

In certain scenarios, combining EasyAR Sense with AR Foundation may be required to leverage additional functionalities (e.g., plane detection on specific devices) and interfaces offered by AR Foundation. In such cases, EasyAR Sense interacts with the Unity engine through interfaces provided by AR Foundation.

However, since EasyAR delivers more features and broader device compatibility than system-level AR, using AR Foundation alone generally cannot achieve the same results as EasyAR.

Components of session

A typical session mainly consists of the following components:

  • frame source: provides physical camera images and sensor data. Sometimes these components also provide motion tracking data. For example, CameraDeviceFrameSource and MotionTrackerFrameSource
  • frame filter(s): components that provide specific AR functionalities, such as ImageTrackerFrameFilter
  • camera: the virtual camera object in the scene
  • origin: the origin object for motion tracking
Note

In the concept of AR Foundation, motion tracking is considered a mandatory feature, so it always provides an origin.
In the EasyAR system, motion tracking is an optional feature, so the origin is also optional.

[Optional] session data flow

Data flow is one of EasyAR Sense's core concepts. It doesn't affect AR application development in Unity. This section is for those wanting deeper understanding of session mechanics.

In Unity, a session typically represents an EasyAR Sense data flow.

flowchart LR
  S[Frame Source]
  R[Input Frame Recorder<br>Video Input Frame Recorder]
  ift[iFrameThrottler]
  iff[iFrameFork]
  i2f[i2FAdapter]
  fb[fbFrameFork]  
  i2o[i2OAdapter]

  FOT[Object Tracker]
  FIT[Image Tracker]
  FMT[Mega Tracker]
  FSSM[Sparse Spatial Map]
  FST[Surface Tracker]
  FDS[Dense Spatial Map]
  FCR[Cloud Recognizer]
  
  ofj[oFrameJoin]
  off[oFrameFork]
  ofb[oFrameBuffer]

  O(( ))
  ODS(( ))
  OCR(( ))

  S ==> R ==> ift ==> iff
  iff --> i2f
  i2f --> fb
  fb -.-> FOT -.-> ofj
  fb -.-> FIT -.-> ofj
  iff ==> i2o ==> ofj ==> off ==> ofb ==> O
  iff -.-> FMT -.-> ofj
  iff -.-> FSSM -.-> ofj
  iff -.-> FST -.-> ofj
  iff -.-> FDS -.-> ODS
  iff -.-> FCR -.-> OCR
  off --> i2f
  ofb --> ift

This data flow is created during session initialization. Connections beyond the bold data pathways depend on AR components enabled during startup.

Thus, modifying session components allows flexible restructuring of data flow and simultaneous activation of multiple AR features. Implementation details follow in subsequent sections.

Session flow

flowchart LR
  i[Initialize]
  a[Assemble]
  starta[StartSession(Assembled)]
  start[StartSession]
  update((update))
  stop[StopSession]
  di[Deinitialize]

  i --> a --> starta --> update --> stop --> di
  i --> start --> update

  • Initialization
    Initialization activates EasyAR Sense using a license key. Before initialization, only a minimal subset of EasyAR Sense interfaces are available. AR functionality becomes active only after initialization.

  • Assembling
    The assembly process selects appropriate components from the scene based on configuration options and connects them into a cohesive working unit. This process typically completes automatically during startup but can also be manually invoked before startup. After assembly completes, starting an assembled session skips the assembly phase for faster initialization.
    Assembly also serves to verify AR component/input source availability and selects the optimal input source from candidates. This step can determine whether the session can run on the current device.

    Assembly occurs in two phases:

    1. Phase one initiates device support list updates and begins assembly after a fixed wait time. If the device support list completes during this wait, assembly finishes immediately.
    2. Otherwise, phase two executes after device support list completion. If available frame sources transition from unavailable (phase one) to available, and the session failed to start after phase one, it attempts restart.

    Regardless of device list completion status in phase one, the session proceeds after phase one completion.

  • Startup
    Startup initiates AR functionality execution. Before startup, AR components don't process data. After successful startup, the session controls movement of scene objects and manages physical camera image rendering (for certain input sources).

  • Update
    Executes per frame within Unity's rendering loop. The update process modifies the virtual camera (for certain input sources), origin, and tracked targets' transforms based on AR functionality results. Update timing varies across devices but always occurs before rendering.

  • Stop
    Terminates AR functionality execution. Scene objects cease being controlled by the session, and input source data processing halts.

  • Deinitialization
    Releases global resources (without unloading dynamic libraries). AR components become unusable after deinitialization.

Note

All AR functionality can only be used after ARSession.StartSession.

Default lifecycle of session

flowchart LR
  uload("BeforeSceneLoad")
  ustart("MonoBehaviour.Start")
  udestroy("MonoBehaviour.OnDestroy")
  oi{Initialize<br>OnStartup}
  ostart{AutoStart}
  i[Initialize<br>Initialize]
  start[Start<br>StartSession]
  update((Update<br>update))
  stop[Stop<br>StopSession]
  
  uload -.-> ustart -.-> udestroy
  uload --> oi -. true .-> i
  ustart --> ostart -. true .-> start
  udestroy --> stop
  i --> start --> update --> stop

The session lifecycle is generally determined by interface call timing. With default settings, the session is automatically executed at the following points:

Session state

ARSession.State describes the state of a session. A session has the following states:

State Description
None Initial state, session is not started or assembling
Broken Session is broken due to reasons like assembly failure
Assembling In assembly process, which typically may last several frames
Assembled Assembly completed successfully but not yet started
Ready Session started successfully, this state only lasts one frame
Running Session is running
Paused Session is paused

Typically the session state changes when calling interfaces like start and stop. During operation, if serious errors occur, the session may also enter the Broken state. A session in the Broken state cannot be recovered and must be restarted by calling stop first.

You can understand whether the current session is in an available state through its session state. Most functionalities are only available when in the Ready or Running state.

Motion tracking status

ARSession.TrackingStatus describes the motion tracking status of the session, indicating the quality of the device's motion tracking. It has the following states:

Status Description
Optional<MotionTrackingStatus>.Empty Motion tracking is not enabled or the session is not running
NotTracking Motion tracking results are unavailable, possibly due to initialization, tracking loss, or relocalization
Limited Motion tracking is valid but results are suboptimal, potentially due to weak scene texture or excessive motion
Tracking Motion tracking quality is good
Note

In AR Foundation's concept, motion tracking is considered a required feature, so its tracking status is merged with the session state. In the EasyAR system, motion tracking is an optional feature, so its tracking status exists independently and may be empty.

Where is the tracking status of other AR features

Since AR features may track multiple objects simultaneously, the image tracking status and the tracking status of other AR features are not in the session, but within the tracked target components.

You can use TargetController.IsTracked to determine if a tracked target is in the tracking state, or use the TargetController.TargetFound and TargetController.TargetLost events to adjust application content logic when the tracking state changes.

Next steps

Creation

Controlling execution

Accessing components and results

Component reference