AR-driven 3D rendering
AR application development needs to solve a fundamental problem: rendering AR content. This article uses planar image tracking as an example to describe the basic modules, workflow, and rendering implementation of an AR application.
Typical AR application workflow
A typical AR application usually recognizes specific images, objects, or scenes from camera images, tracks their position and pose, and renders virtual content (3D models) according to that position and pose.
For example, the figure above shows an AR application for planar image tracking.
The application workflow is shown below.
flowchart TD
CameraDevice[Camera Device]
Tracker[Tracker]
Renderer[Renderer]
CameraDevice -->|Image Frame| Tracker
Tracker -->|Image Frame + Tracked Pose| Renderer
The workflow contains the following modules.
| Module | Role |
|---|---|
| Physical camera | Provides a sequence of input image frames. An image frame includes the image, the timestamp when the image was generated, and sometimes the camera's position and pose in space |
| Tracker | Computes the position and pose of the tracking target from image frames. Different tracking targets use different trackers, such as planar image trackers and 3D object trackers |
| Renderer | Renders the camera image and the 3D model corresponding to the tracked object onto the screen. On some AR glasses, the camera image may not be rendered, and only the 3D model is rendered |
Rendering on phones
Rendering on phones consists of two parts: rendering the camera image and rendering virtual objects.
Rendering the camera image
There are several parameters to pay attention to when rendering the camera image.
Scaling mode
Usually the camera image needs to fill the entire screen or a window. In this case, you need to handle mismatched aspect ratios between the camera image and the screen/window.
If the camera image center is aligned with the screen/window center and the aspect ratio is kept unchanged, there are two common scaling modes: fit and fill.
Scaling mode Effect Fit Shows all content on the screen, but leaves black bars on the left and right or top and bottom Fill Leaves no black bars, but crops part of the image on the left and right or top and bottom Camera image rotation
On phones, the image recorded by the physical camera is usually fixed relative to the device body and does not change with the screen display orientation. However, changes in the phone body's orientation affect how we define up, down, left, and right for the image. During rendering, the current screen display orientation also affects the orientation of the displayed image.
Usually, rendering needs to determine a rotation angle of the camera image relative to the screen display orientation.
Camera image flipping
In some cases, the front-facing camera is used. At that time, the image usually needs to be flipped horizontally so that it looks like a mirror.
Rendering virtual objects
When rendering virtual objects on a phone, the virtual objects need to align with the camera image. This requires placing both the rendering camera and the objects in a virtual space that fully corresponds to the real space, and rendering with the same field of view and aspect ratio as the physical camera. The camera image and the virtual objects undergo identical perspective projection transforms, except that most of the camera image's perspective projection happens inside the physical camera, while the perspective projection transform for virtual objects is entirely a computation process.
Rendering on headsets
Rendering on headsets differs from rendering on phones and needs to be divided into two cases.
VST
Video See-Through refers to an AR technology where a headset captures images through physical cameras, then displays the camera images and virtual content on the headset screen. A typical example is Vision Pro. Usually, the perspective projection matrices of the camera images and virtual content are set by the SDK provided by the headset, and external code only needs to set the position and pose of virtual content. The physical camera used for tracking and the camera image rendered on the screen may be at different positions, and coordinate transforms are applied during rendering.
OST
Optical See-Through refers to an AR technology where the headset screen is transparent and the headset displays only virtual content on the screen. A typical example is HoloLens. Usually, the perspective projection matrix of virtual content is set by the SDK provided by the headset, and external code only needs to set the position and pose of virtual content. The physical camera used for tracking and the camera image rendered on the screen may be at different positions, and coordinate transforms are applied during rendering.
Platform-specific guides
AR-driven 3D rendering is closely related to the platform. Refer to the following guides for development based on your target platform: