Choose a 3D engine before augmented reality development
The first step in AR development is choosing a suitable 3D engine. This chapter explains why a 3D engine is needed, common 3D engines used in AR development, and their respective advantages and disadvantages.
Why AR needs a 3D engine
Augmented reality is not simply overlaying 2D or 3D images on the camera view. It is a real-time 3D system whose core capabilities may also include:
Real camera modeling
To make rendered virtual objects look more realistic in the real world, the projection matrix of the camera in the 3D engine used for rendering must be adjusted according to the parameters used by the actual camera image, such as intrinsics, extrinsics, and distortion model.
Spatial coordinate system management
Unified management of the position and pose of devices, environments, AR content, and other elements, including selection, setup, and conversion among world coordinates, camera coordinates, and device coordinates.
Real-time 3D rendering
Implement realistic occlusion between virtual objects and the environment based on real-time estimated scene depth or reconstructed mesh, and simulate shadows based on lighting estimation algorithms to achieve realistic fusion of virtual and real content.
Resource and lifecycle management
Manage virtual AR resources and content, covering lifecycle management such as loading, presentation, and unloading.
These capabilities form the core responsibilities of a typical 3D engine. Therefore, choosing an appropriate 3D engine according to specific project requirements is one of the necessary prerequisites for quickly implementing AR effects.
Common 3D engines
EasyAR supports multiple 3D engines, including common 3D engines such as Unity, Unreal, and native development. EasyAR provides samples and development documentation for Unity and Native.
Unity
Unity is positioned as a general-purpose real-time 3D engine and is currently the first choice for most AR developers. Unity natively supports cross-platform development for Windows/macOS and iOS/Android/visionOS. The Unity ecosystem is mature, with complete documentation and samples.
Native
Compared with using high-level packaged engines such as Unity, developing AR directly with native graphics APIs such as OpenGL, Vulkan, and Metal has the following advantages: fewer system dependencies, an extremely streamlined runtime environment, and deep customization of camera models and underlying algorithms. Native API development has high engineering and maintenance costs, lacks mature editors and debugging tools, has low iteration efficiency, and is difficult to use cross-platform. It is not conducive to rapid product-level delivery and is usually used for implementing simple features.
Web
Web requires no installation and can be used directly in a browser, making distribution and user reach extremely low cost. It is naturally cross-platform, suitable for rapid launch and large-scale user access, and has a relatively low development threshold with a mature frontend ecosystem.
At present, Web is still relatively limited in AR applications. The main limitations are that performance is constrained by browsers and security sandboxes, support for core AR capabilities such as motion tracking, occlusion, and accurate lighting is insufficient, access to device capabilities is limited, and stability and consistency are difficult to guarantee. Therefore, Web AR is suitable for lightweight presentation and marketing, but not for high-precision, highly interactive complex AR applications.