Choosing a 3D engine before starting augmented reality development
The first step in AR development is selecting the right 3D engine. This chapter explains why a 3D engine is needed, the common 3D engines for AR development, and their respective pros and cons.
Why AR needs a 3D engine
Augmented reality is not simply about overlaying 2D or 3D images on a camera feed. It is a real-time 3D system, and its core capabilities may also include:
Real camera modeling
To make rendered virtual objects appear more realistic in the real world, the projection matrix of the 3D engine's camera used for rendering needs to be adjusted based on the parameters (e.g., intrinsic and extrinsic parameters, distortion models, etc.) of the actual camera feed.
Spatial coordinate system management
Unified management of the positions and poses between devices, environments, AR content, etc., handling the selection, setup, and transformation of world coordinates, camera coordinates, and device coordinates.
Real-time 3D rendering
Achieve realistic occlusion effects between virtual objects and the environment based on real-time estimated scene depth or reconstructed meshes, simulate shadows using lighting estimation algorithms, and achieve lifelike virtual-real fusion.
Resource and lifecycle management
Manage virtual AR resources and content, covering their loading, display, unloading, and other lifecycle aspects.
These capabilities form the core responsibilities of a typical 3D engine. Therefore, selecting the right 3D engine based on specific project requirements is one of the necessary prerequisites for quickly achieving AR effects.
Common 3D engines
EasyAR supports multiple 3D engines, including common ones like Unity, Unreal, or native development. EasyAR provides samples and documentation for Unity and Native.
Unity
Unity is positioned as a general-purpose real-time 3D engine and is currently the first choice for most AR developers. Unity natively supports cross-platform development for Windows/macOS as well as iOS/Android/visionOS. The Unity ecosystem is mature, with comprehensive documentation and examples.
Native
Compared to using high-level engines like Unity, developing AR directly based on native graphics APIs (e.g., OpenGL, Vulkan, Metal) offers the following advantages: fewer system dependencies, an extremely streamlined runtime environment, and deep customization of camera models and underlying algorithms. However, native API development comes with high engineering and maintenance costs, lacks mature editors and debugging tools, has low iteration efficiency, and faces significant challenges in cross-platform development, making it unsuitable for rapid product delivery. It is typically used for implementing simple functionalities.
Web
Web-based AR requires no installation and can be used directly in a browser, with extremely low distribution and reach costs. It is inherently cross-platform and suitable for rapid deployment and large-scale user access. The development barrier is relatively low, and the front-end ecosystem is mature.
Currently, Web-based AR is still quite limited, mainly due to performance constraints imposed by browsers and security sandboxes, insufficient support for core AR capabilities like motion tracking, occlusion, and precise lighting, restricted access to device capabilities, and difficulties in ensuring stability and consistency.
Therefore, Web AR is suitable for "lightweight displays and marketing" but not for high-precision, highly interactive complex AR applications.