Table of Contents

Combining Plane Tracking and Motion Tracking

This article describes how to fuse plane image tracking with device motion tracking to improve tracking stability and user experience in complex scenarios. It covers core principles, expected effects, and analysis of potential issues.

Basic Principle

Motion Fusion combines pose data from plane image tracking and device motion tracking to achieve more robust pose estimation. The core workflow is as follows:

Data Synchronization and Complementarity

  • Visual tracking: Calculates the pose (position + rotation) of the current frame through image feature point matching, but is easily affected by occlusion, blur, or fast movement.
  • Motion tracking: Uses high-frequency IMU sensor output and visual image output to obtain device motion data, but cumulative drift errors may occur.
  • Motion Fusion:
    • Align the coordinate systems of the visual tracking pose and the device motion tracking pose.
    • When the target image is clearly visible and moves stably: visual tracking is dominant. The visual tracking pose is continuously fed into the fusion module for correction, reducing cumulative drift of the entire system.
    • When the target image is lost, occupies too small a portion of the image, or moves quickly: visual tracking fails and motion tracking becomes dominant. The fused pose is predicted based on the current motion tracking pose.

Key Technical Points

  • Timestamp alignment: Align timestamps of visual frames with motion tracking data to avoid jitter caused by latency.
  • Coordinate system alignment: Align coordinate systems based on the visual tracking trajectory and the motion tracking trajectory.
  • Relocalization: When the image appears again, visual tracking takes over and quickly corrects possible cumulative errors, pulling the virtual object back to the correct position.

Applicable Scenarios and Limitations

Motion Fusion is not suitable for all scenarios. Motion Fusion is not applicable in any of the following cases:

  • The target device does not support motion tracking features such as ARCore/ARKit. For the detailed device support list, see Motion Tracking Device Support.
  • The target image/plane object is dynamic in the scene, such as a card held in the hand.

In other scenarios, Motion Fusion greatly improves the user experience of plane image tracking, including but not limited to:

  • Fast movement: Fast handheld device movement may cause motion blur and make visual tracking fail.
  • Target disappears: When the view leaves the target itself or the target is occluded by a dynamic object such as a pedestrian, virtual content in the entire scene can still be presented.
  • Far from target: When the user moves the handheld device away and the target image occupies too small a portion of the image, tracking can still continue stably.
  • Low-light conditions: Visual tracking performance decreases, and the experience needs to be maintained.

Effects and Expected Results

When the scenario is applicable, using Motion Fusion provides a more stable and smoother user experience than using plane image tracking alone.

Ideal Results

  • More stable tracking: Virtual objects do not jitter or jump.
  • Smooth transition: When visual tracking fails, changes in the fused pose remain continuous and natural.
  • Interference resistance: When the target image is lost or occluded, or the device moves quickly, the virtual object can still follow device motion and continue tracking.

Non-Ideal Cases and Responses

Phenomenon Cause User perception Solution
Not effective initially Motion tracking takes time to initialize Content disappears in the initial stage Provide appropriate UI prompts and ensure system motion tracking initialization is complete
Obvious drift System errors accumulate, with no visual correction for a long time Virtual objects deviate from the original position Guide users to shorten occlusion time, or add visual relocalization prompts
Performance degradation Two features run simultaneously for a long time Frame rate drops and the image stutters This is normal; Motion Fusion can be disabled through the API

Expected Result Verification Method

Test with a supported device in a real scene:

  1. Aim at the image and confirm that the virtual object is stable.
  2. Occlude the image with your hand for 2 seconds and move the device, and observe whether the virtual object moves smoothly.
  3. Remove the hand and confirm that the virtual object quickly returns to the correct position without jumping.

Summary and Best Practices

Motion Fusion significantly improves the robustness of plane image tracking in many scenarios, but it requires device hardware support and sufficient performance. Developers should selectively enable this feature according to target user devices, and provide fallback solutions on low-performance devices.

API reference for enabling/disabling Motion Fusion in real time: