Table of Contents

Concepts and Workflow of MegaTracker

This article introduces the basic concepts of MegaTracker and the relationship between MegaTracker, WeChat's native AR system VisionKit, and the rendering framework xr-frame.

Before You Begin

Learn the following through Mega Overview:

  • The basic principles of Mega localization and tracking.
  • What a Mega Block is.
  • The expected result after integrating Mega.

What Is a Plane AR Tracker

The Plane AR Tracker in xr-frame is essentially a wrapper around VisionKit 6DoF plane capability.

After isARCamera is enabled on the xr-frame camera component, the camera's 3D transform is synchronized with the AR system (VisionKit) every frame.

xr-frame provides 3D rendering capabilities, while VisionKit provides motion tracking capabilities in the real-world spatial coordinate system.

The Plane AR Tracker cannot be used together with other AR trackers.

What Is MegaTracker

MegaTracker is the core algorithm component that connects the WeChat AR system (VisionKit) with the Mega spatial computing service. It provides cloud localization.

  • Input: the camera pose in the VisionKit coordinate system calculated by VisionKit for each frame (that is, 6DoF data), and the camera image of the frame used for Mega localization.

  • Output: the camera pose under the currently localized and tracked Mega Block.

How MegaTracker Works on xr-frame

flowchart BT
    subgraph Using xr-frame Only
        direction BT
        PlaneARTracker_1[PlaneARTracker] -->|MotionData & Image| XRFrame_1[xr-frame]
    end

    subgraph Using Mega Plugin
        direction BT
        PlaneARTracker_2[PlaneARTracker] -->|MotionData & Image| MegaTracker
        MegaTracker -->|CameraTransform| XRFrame_2[xr-frame]
    end
  • In the data flow provided natively by WeChat, the xr-frame camera component is updated every frame directly from the result of the Plane AR Tracker.
  • In the data flow provided by the Mega mini program, the camera pose in the VisionKit coordinate system (that is, 6DoF data) and the image data of the localization frame are input to MegaTracker. After cloud localization and local computation, it outputs the camera pose under the currently localized and tracked Mega Block, and finally updates the LocalTransform of the camera under the Mega Block node in the xr-frame scene. At this point, MegaTracker takes control of the camera, and xr-frame no longer updates the camera according to the AR tracker.

MegaTracker depends deeply on the 6DoF motion data provided by the plane tracker. Therefore, MegaTracker cannot start working before the plane tracker finishes initialization and establishes a stable tracking state. In addition, AR tracking stability is limited by environmental features. In extreme scenarios such as large textureless areas (for example, white walls) or long camera occlusion, if the underlying WeChat plane tracking drifts or is lost, MegaTracker will also enter a failed state because it loses a reliable input source.

Next Steps