Table of Contents

Get session runtime results

During session runtime, the transforms of some objects in the scene are modified, along with changes to the camera's view. Sometimes, these modifications may not fully meet application requirements, necessitating access to session results per frame for secondary processing. This article explains how to obtain and utilize these result data.

Before starting

Get InputFrame updates

Use the InputFrameUpdate event to obtain InputFrame updates. This event triggers only when the InputFrame in session's per-frame output data changes.

Note

InputFrameUpdate is only valid in sessions where rendering is handled by EasyAR. Generally, it's invalid when using AR Foundation or HMDs; use methods provided by those third-party libraries to get data updates instead.

InputFrame provides access to physical camera images, camera parameters, timestamps, physical camera transforms relative to the world coordinate system, and tracking status. However, since camera transforms are already applied by the session to virtual cameras and other objects, accessing camera transforms through InputFrame is usually unnecessary.

Get current frame's physical camera image

Use the InputFrame.image() method to obtain Image type physical camera image data.

For example, this code captures the physical camera image during InputFrame updates:

Session.InputFrameUpdate += (inputFrame) => {
    using (var image = inputFrame.image())
    {
    }
};
Caution

When using Image type data or other class-type data derived from it, ensure Dispose() is correctly called (handled by the using statement above). Otherwise, memory leaks or even frozen rendering may occur.

If retaining InputFrame or Image for future frames is needed, increase the value of ARAssembly.ExtraBufferCapacity based on data volume. Otherwise, buffer shortages may cause data retrieval failures.

To retain InputFrame, call Clone() to create a reference copy, then call Dispose() on the copy when no longer needed.

Since physical camera frame rates are typically lower than rendering frame rates, InputFrameUpdate won't trigger every rendering frame. However, physical camera image rendering also doesn't update every frame. All rendering frames between InputFrameUpdate events display the same image as the current InputFrame.

Note

Images in InputFrame always match the virtual camera background view, but the rendered background may be scaled or cropped. Thus, size or aspect ratio discrepancies between obtained images and screen displays are normal.

Note: InputFrame.image() returns CPU-readable image data, not GPU textures. For GPU usage, upload image data to textures or directly obtain GPU textures via CameraImageRenderer.RequestTargetTexture(Action<Camera, RenderTexture>).

[Optional] Intercept physical camera image rendering

Use ARAssembly.CameraImageRenderer to control physical camera image rendering.

This code stops physical camera image rendering:

if (Session.Assembly != null && Session.Assembly.CameraImageRenderer.OnSome)
{
    Session.Assembly.CameraImageRenderer.Value.enabled = false;
}

First check if ARAssembly.CameraImageRenderer exists.

Note

The above method only works in sessions rendered by EasyAR. It's generally invalid for AR Foundation or HMDs; use third-party library methods for equivalent functionality instead.

After stopping physical camera rendering, applications can use InputFrame to obtain physical camera images for custom rendering.

Get transform updates

Access transform data of scene objects after per-frame session updates via the PostSessionUpdate event.

Note

For certain features (e.g., Mega), AR computation runs every rendering frame even without image changes or explicit update requests. Thus, to capture all transform changes, per-frame data retrieval is essential.

Get virtual camera transform

Access camera transform through ARAssembly.Camera:

Session.PostSessionUpdate += () =>
{
    var position = Session.Assembly.Camera.transform.position;
    var rotation = Session.Assembly.Camera.transform.rotation;
};

Get target transform

Access target transform via the specific target object in use. For image tracking, this is the object containing the ImageTargetController component.

Session.PostSessionUpdate += () =>
{
    var position = target.transform.position;
    var rotation = target.transform.rotation;
};

[Optional] Get pose

Pose is a data structure describing an object's position and orientation, typically comprising position and rotation. In AR, pose usually describes physical cameras or tracked targets relative to a reference coordinate system.

Unity doesn't provide raw pose data because pose is generally used to drive object movement—handled automatically by sessions. For content computation and rendering, transforms suffice.

Important

Before proceeding, reconsider whether transforms of cameras and tracked targets in the scene already meet requirements. Typically, additional pose data is unnecessary.

If pose data is absolutely required, calculate it from transforms during PostSessionUpdate. Usually, the relative transform between target and camera obtained here is the pose.

This code demonstrates obtaining camera/target transforms and computing their relative pose:

Session.PostSessionUpdate += () =>
{
    Pose cameraToWorld = new(Session.Assembly.Camera.transform.position, Session.Assembly.Camera.transform.rotation);
    Pose targetToWorld = new(target.transform.position, target.transform.rotation);
    Pose worldToTarget = new()
    {
        position = Quaternion.Inverse(targetToWorld.rotation) * (-targetToWorld.position),
        rotation = Quaternion.Inverse(targetToWorld.rotation)
    };
    Pose cameraToTarget = cameraToWorld.GetTransformedBy(worldToTarget);
};
Caution

If using AR Foundation, HMDs, or other third-party libraries that modify scene camera transforms, ensure pose calculations occur after their update logic completes. Otherwise, results may be incorrect. In such scenarios, the relative pose between target and origin in PostSessionUpdate remains accurate.

[Optional] Intercept transform updates

During AR operation, transforms of Unity cameras and tracked targets are automatically updated by sessions to ensure rendering correctness and consistency. There's no direct method to intercept these updates.

To customize transform update logic, use PostSessionUpdate with this approach:

  1. Detach objects requiring custom updates from session-controlled hierarchies (they shouldn't be child nodes).
  2. Record these objects' transforms during PostSessionUpdate.
  3. Apply custom transform update logic using session-provided data within the same event.
Note

Using PostSessionUpdate is mandatory because sessions only stop modifying scene objects after this point.

This method cannot modify cameras—camera customization requires more complex logic.

Note: This only customizes non-session-controlled objects. Session-controlled transforms will be overwritten next frame if externally modified, potentially affecting computational accuracy.

Caution

You must ensure transform correctness when using this method; otherwise, AR rendering may fail.

If using AR Foundation, HMDs, or other third-party libraries, ensure their update logic doesn't conflict with custom logic.