Table of Contents

Unity AR Entry Point —— AR Session

The AR session is the entry point for all AR functionalities. Through the following content, you will learn about the basic concepts, composition, and workflow of an AR session, as well as how it relates to Unity AR Foundation's AR Session. You will also understand how the data flow of EasyAR Sense works in Unity.

What is an AR session

All AR processes (such as object tracking) are executed within the native library, specifically EasyAR Sense. The session is the primary entry point for AR functionality in Unity. It manages the operation process and state of the AR system, including reading data from the physical camera and sensors, analyzing the real world, driving the movement and rendering of virtual cameras and other objects in the scene, among other tasks.

flowchart LR
  A((Image<br>and other data))
  B[Session]
  C([Camera])
  O([Origin])
  T([Target])
  A --> B
  B -. transform .-> C
  B -. transform .-> O
  B -. transform .-> T

[Optional] EasyAR session and AR Foundation session

EasyAR session is the core component for using EasyAR in Unity and can operate independently of any third-party or system AR features. On the other hand, AR Foundation session is part of the Unity XR framework and can only utilize features provided by Unity XR plugins (such as ARKit or ARCore).

flowchart TD  
  A1[EasyAR<br>AR Session]  
  A2[EasyAR Sense]  
  A1 --> A2  

  B1[AR Foundation<br>AR Session]  
  B2[ARKit Plugin]  
  B3[ARCore Plugin]  
  B1 --> B2  
  B1 --> B3

When using EasyAR, it is usually not necessary to install and use AR Foundation simultaneously. For example, features like image tracking and motion tracking are independently provided by EasyAR Sense.

In some cases, it may be necessary to combine EasyAR Sense with AR Foundation to leverage additional features (such as plane detection on certain devices) and interfaces provided by AR Foundation. In such scenarios, EasyAR Sense interacts with the Unity engine through the interfaces provided by AR Foundation.

However, since EasyAR offers more features and better device compatibility than system AR, using AR Foundation alone typically cannot achieve the same results as EasyAR.

Composition of a session

A typical session mainly consists of the following components:

  • frame source: provides physical camera images and sensor data, sometimes these components also provide motion tracking data. For example, CameraDeviceFrameSource and MotionTrackerFrameSource
  • frame filter(s): components that provide specific AR functionalities, such as ImageTrackerFrameFilter
  • camera: a virtual camera object in the scene
  • origin: the origin object for motion tracking
Note

In the concept of AR Foundation, motion tracking is considered a mandatory feature, so it always provides an origin.
In the EasyAR system, motion tracking is an optional feature, so the origin is also optional.

[Optional] Data flow of session

Data flow is one of the core concepts of EasyAR Sense. It does not affect your development of AR applications in Unity. If you want to gain a deeper understanding of how session works, you can read this section.

In Unity, a session typically represents a data flow of EasyAR Sense.

flowchart LR  
  S[Frame Source]  
  R[Input Frame Recorder<br>Video Input Frame Recorder]  
  ift[iFrameThrottler]  
  iff[iFrameFork]  
  i2f[i2FAdapter]  
  fb[fbFrameFork]    
  i2o[i2OAdapter]  

  FOT[Object Tracker]  
  FIT[Image Tracker]  
  FMT[Mega Tracker]  
  FSSM[Sparse Spatial Map]  
  FST[Surface Tracker]  
  FDS[Dense Spatial Map]  
  FCR[Cloud Recognizer]  

  ofj[oFrameJoin]  
  off[oFrameFork]  
  ofb[oFrameBuffer]  

  O(( ))  
  ODS(( ))  
  OCR(( ))  

  S ==> R ==> ift ==> iff  
  iff --> i2f  
  i2f --> fb  
  fb -.-> FOT -.-> ofj  
  fb -.-> FIT -.-> ofj  
  iff ==> i2o ==> ofj ==> off ==> ofb ==> O  
  iff -.-> FMT -.-> ofj  
  iff -.-> FSSM -.-> ofj  
  iff -.-> FST -.-> ofj  
  iff -.-> FDS -.-> ODS  
  iff -.-> FCR -.-> OCR  
  off --> i2f  
  ofb --> ift

This data flow is created during the session startup process. Except for the bold data paths, whether other parts are connected depends on the AR components enabled during the startup process.

Therefore, by modifying the components enabled in the session, you can flexibly change the structure and functionality of the data flow, and it is also convenient to enable multiple AR features simultaneously. This method will be introduced in detail in the following paragraphs.

Session Flow

flowchart LR
  i[Initialize<br>Initialize]
  a[Assemble<br>Assemble]
  starta["Start (Assembled)<br>StartSession(Assembled)"]
  start[Start<br>StartSession]
  update((Update<br>update))
  stop[Stop<br>StopSession]
  di[Deinitialize<br>Deinitialize]

  i --> a --> starta --> update --> stop --> di
  i --> start --> update

  • Initialize
    Initialization is the process of starting EasyAR Sense using a license key. Before initialization, only a very small portion of EasyAR Sense interfaces can be used. After initialization, AR features will be activated.

  • Assemble
    The assembly process selects suitable components from the scene based on the assembly options configuration and connects them into a complete working unit. This process is usually completed automatically at startup, but it can also be manually called before startup to complete the process. After assembly, you can start the assembled session to skip the assembly process and speed up startup.
    The assembly process also has an important purpose: to determine the availability of AR components and input sources, and to select the most suitable input source from all candidates. This step can also be used to determine whether the current session can run on the current device.

    The assembly process is divided into two stages:

    1. The first stage starts the device support list update and begins assembly after waiting for a fixed time according to the configuration. If the device support list update is completed after the first stage wait, the assembly process ends;
    2. Otherwise, the assembly process enters the second stage, which is executed after the device support list update is completed. In this stage, if the available frame source changes from no available frame source in the first stage to having an available frame source, and the session fails to start after the first stage, it will attempt to restart the session.

    Regardless of whether the device list update is completed in the first stage, the session will continue to execute subsequent steps after the first stage is completed.

  • Start
    Starting is the process of beginning AR feature operation. Before starting, AR feature components will not process any data. After a normal start, the session will begin controlling the movement of some objects in the scene and, when using certain input sources, control the rendering of physical camera images.

  • Update
    The update process is executed every frame in Unity's rendering loop. The update process modifies the transform of the virtual camera (for some input sources), the origin, and the tracked targets based on the current AR feature operation results. The execution time of the update process is not the same on different devices, but it will always be executed before rendering.

  • Stop
    Stopping will terminate the operation of AR features. Objects in the scene will no longer be controlled by the session, and input source data will not be processed.

  • Deinitialize
    Deinitialization releases some global resources (without unloading the dynamic library). After deinitialization, AR feature components will be unavailable.

Note

All AR features can only be used after ARSession.StartSession.

Default lifecycle of session

flowchart LR
  uload("BeforeSceneLoad")
  ustart("MonoBehaviour.Start")
  udestroy("MonoBehaviour.OnDestroy")
  oi{Initialize<br>OnStartup}
  ostart{AutoStart}
  i[Initialize<br>Initialize]
  start[Start<br>StartSession]
  update((Update<br>update))
  stop[Stop<br>StopSession]
  
  uload -.-> ustart -.-> udestroy
  uload --> oi -. true .-> i
  ustart --> ostart -. true .-> start
  udestroy --> stop
  i --> start --> update --> stop

The lifecycle of a session is generally determined by the timing of interface calls. With default settings, the session will automatically execute at the following times:

Session state

ARSession.State describes the state of the session. A session has the following states:

State Description
None Initial state, the session is not started or assembled
Broken The session is broken due to assembly failure or other reasons
Assembling In the process of assembly, which usually may last for several frames
Assembled Successfully completed assembly, but not yet started
Ready The session is successfully started, this state will only last for one frame
Running The session is running
Paused The session is paused

Usually, the state of the session will change when calling interfaces such as start and stop. During operation, if a serious error occurs, the session may also enter the Broken state. A session in the Broken state cannot resume operation and must be restarted after calling stop.

You can understand whether the current session is in an available state through the session state. Most functions can only be used in the Ready or Running state.

Motion tracking status

ARSession.TrackingStatus describes the motion tracking status of the session, indicating the quality of the device's motion tracking. It has the following states:

Status Description
Optional<MotionTrackingStatus>.Empty Motion tracking is not enabled or the session is not running
NotTracking Motion tracking results are unavailable, possibly due to initialization, tracking loss, or relocalization
Limited Motion tracking is valid but the results are not ideal, possibly due to weak texture in the current area or fast motion
Tracking Motion tracking quality is good
Note

In the concept of AR Foundation, motion tracking is treated as a mandatory feature, so its tracking status is merged with the session status.
In the EasyAR system, motion tracking is an optional feature, so the tracking status exists independently and may be empty.

Where is the tracking status of other AR features

Since AR features may track multiple objects simultaneously, the image tracking status and the tracking status of other AR features are not in the session, but in the target-tracking component.

You can use TargetController.IsTracked to check if the tracking target is in the tracking state, or use the TargetController.TargetFound and TargetController.TargetLost events to adjust the application logic when the tracking state changes.

Next steps

Create

Control execution

Access components and results

Component reference