Table of Contents

Object tracking combined with motion tracking

This article introduces how to integrate 3D object tracking with device motion tracking to enhance tracking stability and user experience in complex scenarios. Content includes core principles, expected effects, and potential issue analysis.

Basic principles

Motion fusion combines pose data from 3D object tracking and device motion tracking to achieve more robust pose estimation. The core process is as follows:

Data synchronization and complementarity

  • Visual tracking: Calculates current frame pose (position + rotation) through image feature point matching but is susceptible to occlusion, blurring, or rapid movement.
  • Motion tracking: Utilizes IMU sensor high-frequency output and visual image data to obtain device motion data, but suffers from cumulative drift errors.
  • Fusion mechanism:
    • Aligns coordinate systems between visual tracking poses and motion tracking poses.
    • When the target object is clearly visible with stable motion: Prioritizes visual tracking. Continuously feeds visual tracking poses into the fusion module for correction to reduce system-wide cumulative drift.
    • When the target object is lost, occupies minimal screen space, or moves rapidly: Visual tracking fails, prioritizing motion tracking. Predicts fused poses based on current motion tracking data.

Key technical points

  • Timestamp alignment: Aligns visual frame timestamps with motion tracking data to prevent jitter caused by latency.
  • Coordinate system alignment: Aligns coordinate systems based on trajectories from visual tracking and motion tracking.
  • Relocalization: When the target object reappears, visual tracking takes over to quickly correct potential cumulative errors, "pulling" virtual objects back to correct positions.

Applicable scenarios and limitations

Motion fusion is not suitable for all scenarios. It does not apply under the following circumstances:

  • Target devices don't support motion tracking features like ARCore/ARKit. Refer to device compatibility list: Motion tracking device support.
  • Target objects are dynamic in the scene, e.g., handheld toys or figurines.

In other scenarios, motion fusion significantly enhances user experience for 3D object tracking, including but not limited to:

  • Rapid motion: Device moves quickly causing motion blur that invalidates visual tracking.
  • Target disappearance: Maintains virtual content when target leaves the frame or gets occluded by dynamic objects (e.g., pedestrians).
  • Distance from target: Maintains stable tracking when target occupies minimal screen space due to device distance.
  • Low-light conditions: Sustains experience when visual tracking performance degrades.

Effects and expected results

When applicable, motion fusion delivers more stable and smoother user experience than pure 3D object tracking.

Ideal effects

  • More stable tracking: Virtual objects show no jitter or sudden jumps.
  • Smooth transition: Pose changes remain continuous and natural during visual tracking failures.
  • Anti-interference: Virtual objects persist in tracking device motion during target loss/occlusion or rapid device movement.

Suboptimal situations and solutions

Phenomenon Cause User perception Solution
Initial failure Motion tracking requires initialization time Content disappears during startup Provide UI prompts until motion tracking initializes
Significant drift Accumulated system errors without visual correction Virtual object deviates from position Guide users to reduce occlusion duration or add relocalization cues
Performance drop Long-term concurrent operation of both features Frame rate drops or stutters Expected behavior; disable motion fusion via API when needed

Validation method for expected results

Test with compatible devices in real scenes:

  1. Align with target object, confirm virtual object stability.
  2. Occlude object for 2 seconds while moving device, observe smooth virtual object movement.
  3. Remove occlusion, confirm virtual object snaps back quickly without jumps.

Summary and best practices

Motion fusion significantly enhances 3D object tracking robustness in many scenarios but requires hardware support and sufficient device performance. Developers should selectively enable this feature based on target user devices and provide fallback solutions for low-performance devices.

API references for real-time motion fusion control: