Table of Contents

Creating an image input extension

Before you begin

Creating an external frame data source class

Inherit from ExternalImageStreamFrameSource to create an image input extension. It is a subclass of MonoBehaviour, and the filename should match the class name.

For example:

public class MyFrameSource : ExternalImageStreamFrameSource
{
}

The sample Workflow_FrameSource_ExternalImageStream is an implementation of an image input extension using a video recorded on a mobile phone with ARCore as input. This video was captured using ARCore on a Pixel 2 via camera callback (not screen recording).

Device definition

Override IsCameraUnderControl and return true.

Override IsHMD to define whether the device is a head-mounted display.

For example, set to false when using video as input.

protected override bool IsHMD => false;

Override Display to define the device display.

For example, if running only on mobile phones, use Display.DefaultSystemDisplay whose rotation value automatically changes according to the current display state of the operating system.

protected override IDisplay Display => easyar.Display.DefaultSystemDisplay;

Availability

Override IsAvailable to define whether the device is available.

For example, when using video as input, it is always available:

protected override Optional<bool> IsAvailable => true;

If IsAvailable cannot be determined during session assembly, override the CheckAvailability() coroutine to block the assembly process until availability is confirmed.

Virtual camera

Override Camera to provide a virtual camera.

For instance, Camera.main may be used as the virtual camera for the session:

protected override Camera Camera => Camera.main;

Physical camera

Override DeviceCameras with the FrameSourceCamera type to provide device physical camera information. This data is used when inputting camera frame data. Must be completed when CameraFrameStarted is true.

For example, using the video from the sample Workflow_FrameSource_ExternalImageStream:

private FrameSourceCamera deviceCamera;
protected override List<FrameSourceCamera> DeviceCameras => new List<FrameSourceCamera> { deviceCamera };

{
    var size = new Vector2Int(640, 360);
    var cameraType = CameraDeviceType.Back;
    var cameraOrientation = 90;
    deviceCamera = new FrameSourceCamera(cameraType, cameraOrientation, size, new Vector2(30, 30));
    started = true;
}
Caution

Several input parameters here need to be set according to the actual video used. The parameters in the code above are only applicable to the sample video.

Override CameraFrameStarted to provide the identifier for when camera frame input starts.

For example:

protected override bool CameraFrameStarted => started;

Session start and stop

Override OnSessionStart(ARSession) and perform AR-specific initialization work. Ensure to call base.OnSessionStart first.

For example:

protected override void OnSessionStart(ARSession session)
{
    base.OnSessionStart(session);
    ...
}

This is the appropriate place to open the device camera, especially if these cameras aren't designed to remain constantly active. It's also suitable for obtaining calibration data that won't change throughout the lifecycle. Sometimes it may be necessary to wait for the device to become ready or for data updates before this data can be acquired.

Additionally, this is a suitable location to start the data input loop. You could also write this loop in Update() or other methods, particularly when data needs to be acquired at specific points in Unity's execution order. Do not input data before the session is ready.

If needed, you can skip the startup process and perform data checks during each update - this entirely depends on specific requirements.

For example, when using video as input, you can start video playback and initiate the data input loop here:

protected override void OnSessionStart(ARSession session)
{
    base.OnSessionStart(session);
    ...
    player.Play();
    StartCoroutine(VideoDataToInputFrames());
}

Override OnSessionStop() and release resources, ensuring to call base.OnSessionStop.

For example, when using video as input, you can stop video playback and release related resources here:

protected override void OnSessionStop()
{
    base.OnSessionStop();

    StopAllCoroutines();
    player.Stop();
    if (renderTexture) { Destroy(renderTexture); }
    cameraParameters?.Dispose();
    cameraParameters = null;
    frameIndex = -1;
    started = false;
    deviceCamera?.Dispose();
    deviceCamera = null;
}

Acquire camera frame data from device or file

Images can be obtained from any source—system cameras, USB cameras, video files, networks, etc.—as long as the data can be converted into the format required by Image. Methods to acquire data from these devices or files vary; consult their respective documentation.

For example, when using video as input, use Texture2D.ReadPixels(Rect, int, int, bool) to capture frame data from a video player’s RenderTexture. Then copy the data from Texture2D.GetRawTextureData() into Buffer:

void VideoDataToInputFrames()
{
    ...
    RenderTexture.active = renderTexture;
    var pixelSize = new Vector2Int((int)player.width, (int)player.height);
    var texture = new Texture2D(pixelSize.x, pixelSize.y, TextureFormat.RGB24, false);
    texture.ReadPixels(new Rect(0, 0, pixelSize.x, pixelSize.y), 0, 0);
    texture.Apply();
    RenderTexture.active = null;
    ...
    CopyRawTextureData(buffer, texture.GetRawTextureData<byte>(), pixelSize);
} 

static unsafe void CopyRawTextureData(Buffer buffer, Unity.Collections.NativeArray<byte> data, Vector2Int size)
{
    int oneLineLength = size.x * 3;
    int totalLength = oneLineLength * size.y;
    var ptr = new IntPtr(data.GetUnsafeReadOnlyPtr());
    for (int i = 0; i < size.y; i++)
    {
        buffer.tryCopyFrom(ptr, oneLineLength * i, totalLength - oneLineLength * (i + 1), oneLineLength);
    }
}
Caution

As shown above, data copied from the pointer of Texture2D must be vertically flipped to restore normal image memory arrangement.

While acquiring images, you must also obtain calibration data from the camera (or equivalent) and create an CameraParameters instance.

If the source is a mobile camera callback with uncropped data, use the device’s native calibration data directly. When using ARCore/ARKit APIs, refer to their documentation for camera intrinsics. For image/object tracking, CameraParameters.createWithDefaultIntrinsics(Vec2I, CameraDeviceType, int) can create intrinsics, though this may slightly reduce algorithm efficacy.

For USB cameras, video files, or other non-native sources, calibrate the camera/video frames to obtain correct intrinsics.

Caution

Cropped camera callback data requires recalculating intrinsics. Screen-recorded images typically can’t use native calibration data—calibrate explicitly.

Incorrect intrinsics break AR functionality, causing misalignment of virtual/content, unstable tracking, or frequent failures.

For example, the video in sample Workflow_FrameSource_ExternalImageStream uses these intrinsics and CameraParameters creation process:

var size = new Vector2Int(640, 360);
var cameraType = CameraDeviceType.Back;
var cameraOrientation = 90;
cameraParameters = new CameraParameters(size.ToEasyARVector(), new Vec2F(506.085f, 505.3105f), new Vec2F(318.1032f, 177.6514f), cameraType, cameraOrientation);
Caution

The parameters above only apply to the sample video, as intrinsics were captured alongside it. Always acquire device intrinsics or manually calibrate for other videos/devices.

Input camera frame data

After obtaining camera frame data updates, call HandleCameraFrameData(double, Image, CameraParameters) to input camera frame data.

For example, when using video as input, the implementation is as follows:

IEnumerator VideoDataToInputFrames()
{
    yield return new WaitUntil(() => player.isPrepared);
    var pixelSize = new Vector2Int((int)player.width, (int)player.height);
    ...
    yield return new WaitUntil(() => player.isPlaying && player.frame >= 0);
    while (true)
    {
        yield return null;
        if (frameIndex == player.frame) { continue; }
        frameIndex = player.frame;
        ...
        var pixelFormat = PixelFormat.RGB888;
        var bufferO = TryAcquireBuffer(pixelSize.x * pixelSize.y * 3);
        if (bufferO.OnNone) { continue; }

        var buffer = bufferO.Value;
        CopyRawTextureData(buffer, texture.GetRawTextureData<byte>(), pixelSize);

        using (buffer)
        using (var image = Image.create(buffer, pixelFormat, pixelSize.x, pixelSize.y, pixelSize.x, pixelSize.y))
        {
            HandleCameraFrameData(player.time, image, cameraParameters);
        }
    }
}
Caution

Do not forget to execute Dispose() or release resources through mechanisms like using for Image, Buffer, and other related data after use. Otherwise, severe memory leaks may occur, and buffer pool acquisition might fail.