Table of Contents

Combining planar tracking with motion tracking

This article introduces how to integrate planar image tracking with device motion tracking to enhance tracking stability and user experience in complex scenarios. It covers core principles, expected effects, and potential issue analysis.

Core principles

Motion fusion combines pose data from planar image tracking and device motion tracking to achieve more robust pose estimation. The core workflow is as follows:

Data synchronization and complementation

  • Visual tracking: Calculates the current frame's pose (position + rotation) through image feature point matching, but is susceptible to occlusion, blur, or rapid movement.
  • Motion tracking: Uses IMU sensor high-frequency output combined with visual image output to obtain device motion data, but suffers from cumulative drift errors.
  • Fusion mechanism:
    • Align coordinate systems between visual tracking poses and motion tracking poses.
    • When target images are clearly visible with stable motion: Prioritize visual tracking. Continuously feed visual tracking poses into the fusion module for correction to reduce system drift.
    • When target images are lost, occupy minimal screen space, or during rapid motion: Visual tracking fails; prioritize motion tracking. Predict fusion poses based on current motion tracking poses.

Key technical points

  • Timestamp synchronization: Align visual frame timestamps with motion tracking data to prevent jitter caused by latency.
  • Coordinate system alignment: Align coordinate systems based on trajectories from visual tracking and motion tracking.
  • Relocalization: When images reappear, visual tracking rapidly corrects accumulated errors, "pulling" virtual objects back to correct positions.

Applicable scenarios and limitations

Motion fusion isn't suitable for all scenarios. It won't apply under any of the following conditions:

  • Target devices don't support motion tracking features like ARCore/ARKit. Detailed device support list: Motion tracking device support.
  • Target images/planar objects are dynamic in the scene, e.g., handheld cards during experiences.

In other scenarios, motion fusion significantly enhances user experience for planar image tracking, including but not limited to:

  • Rapid motion: User moves device quickly, causing motion blur that disrupts image tracking.
  • Target disappearance: Maintains virtual content when target leaves view or gets occluded (e.g., by pedestrians).
  • Distance from target: Stabilizes tracking when target occupies minimal screen space due to distance.
  • Low-light conditions: Maintains experience when visual tracking performance degrades.

Effects and expected results

When suitable for the scenario, motion fusion delivers more stable and smoother user experiences than pure planar image tracking.

Ideal effects

  • Stable tracking: Virtual objects remain jitter-free without abrupt jumps.
  • Smooth transitions: Pose changes remain continuous when visual tracking fails.
  • Anti-interference capability: Virtual objects persist during target occlusion/device rapid motion.

Suboptimal scenarios and solutions

Phenomenon Cause User perception Solution
Initial failure Motion tracking requires initialization time Content disappears during startup UI prompts until motion tracking initializes
Significant drift Accumulated system errors without visual correction Virtual objects deviate from position Guide users to reduce occlusion time or add relocalization cues
Performance drop Long-term concurrent operation Frame rate drops/stuttering Normal; disable motion fusion via API when needed

Validation method for expected results

Test with supported devices in real scenarios:

  1. Align device with image; confirm virtual object stability.
  2. Occlude image for 2 seconds while moving device; observe smooth movement.
  3. Remove occlusion; confirm rapid repositioning without jumps.

Summary and best practices

Motion fusion significantly enhances planar image tracking robustness in many scenarios but requires supported hardware and sufficient performance. Developers should selectively enable this feature based on target devices and provide fallback solutions for low-performance devices.

APIs for real-time motion fusion control: