Choosing a 3d engine before augmented reality development
The first step in AR development is selecting a suitable 3d engine. This chapter explains why a 3d engine is needed, common 3d engines for AR development, and their respective advantages and disadvantages.
Why ar needs a 3d engine
Augmented reality is not simply overlaying 2d or 3d images on a camera feed. It is a real-time 3d system whose core capabilities may also include:
- Real camera modeling To make rendered virtual objects appear more realistic in the real world, the projection matrix of the camera in the 3d engine used for rendering needs to be adjusted based on the parameters (e.g., intrinsic and extrinsic parameters, distortion model, etc.) used by the actual camera feed.
- Spatial coordinate system management Unified management of the position and orientation between the device, environment, AR content, etc., responsible for the selection, setting, and conversion between world coordinates, camera coordinates, and device coordinates.
- Real-time 3d rendering Achieve realistic occlusion effects between virtual objects and the environment based on real-time estimated scene depth or reconstructed meshes. Simulate shadows based on light estimation algorithms to achieve realistic virtual-real fusion.
- Resource and lifecycle management Manage virtual AR resources and content, covering their loading, display, unloading, and other lifecycle management.
These capabilities constitute the core responsibilities of a typical 3d engine. Therefore, choosing the right 3d engine based on specific project requirements is one of the necessary prerequisites for rapidly achieving AR effects.
Common 3d engines
EasyAR supports multiple 3d engines, including common ones like Unity, Unreal, or native development. EasyAR provides samples and development documentation for Unity and Native.
Unity
Unity positions itself as a general-purpose real-time 3d engine and is currently the first choice for most AR developers. Unity natively supports cross-platform development for Windows/macOS as well as iOS/Android/visionOS. The Unity ecosystem is mature, with comprehensive documentation and examples.
Native
Compared to using high-level engines like Unity, the advantages of developing AR directly based on native graphics APIs (such as OpenGL, Vulkan, Metal) can be summarized as follows: fewer system dependencies, an extremely streamlined runtime environment, and deep customization capabilities for the camera model and underlying algorithms. However, development using native APIs incurs high engineering and maintenance costs, lacks mature editors and debugging tools, has low iteration efficiency, faces significant challenges in cross-platform development, and is not conducive to rapid product delivery. It is typically used for implementing simple features.
Web
Web-based solutions require no installation and work directly in browsers, offering extremely low distribution and reach costs. They are inherently cross-platform, suitable for rapid deployment and large-scale user access. The development barrier is relatively lower, and the front-end ecosystem is mature.
Currently, Web remains relatively limited in AR applications, mainly reflected in performance constraints imposed by browsers and security sandboxes, insufficient support for core AR capabilities like motion tracking, occlusion, and precise lighting, restricted access to device capabilities, and difficulties in ensuring stability and consistency. Therefore, Web AR is suitable for "lightweight display and marketing" and is not suitable for high-precision, strongly interactive complex AR applications.