Table of Contents

AR scene in unity camera

The rendering of AR effects in Unity relies heavily on cameras. Through the following content, you'll understand the role of cameras in AR scenes and how sessions control camera properties to ensure a proper AR experience.

Before you begin

Role of camera in ar scenes

Cameras in Unity are used to display the game world to players, but their role becomes even more critical in AR scenes. They are not only responsible for rendering virtual content but also need to align with the real world to ensure virtual objects are properly overlaid on real-world scenes.

This video demonstrates a simple AR scene. The left side shows the Scene view, while the right side displays the Game view. The video was recorded in Unity Editor's Play mode using simulated runtime data. The content in the Game view matches what users see in the real world through their mobile devices.

Notice how the camera representing the user (camera icon) moves according to the user's movements in the real world. The white cones capture the trajectory of the camera's position and orientation over time. In the Game view, the camera not only displays virtual content from the Scene view but also overlays real-world imagery beneath the virtual elements, showcasing the typical operation of cameras in AR scenes.

To ensure virtual objects align correctly with real scenes, certain camera properties need adjustment based on AR runtime states. These properties include:

  • Camera transform (position and orientation)
  • Camera field of view (FOV), aspect ratio, and projection matrix
  • Camera culling settings (GL.invertCulling)
Warning

Modifying these properties of the session camera during app development is unsupported. Such changes may cause virtual content to misalign with the real world, negatively impacting user experience. Even if modified through certain methods, the AR system will overwrite these changes during operation, or inconsistent rendering and computational data may lead to unexpected behavior.

Depending on which entity controls these properties, session cameras can be categorized into two types: session-controlled cameras and non-session-controlled cameras.

Session-controlled cameras

If the session's camera isn't managed by any external system (like headsets or AR Foundation), the session automatically controls the aforementioned camera properties to ensure proper alignment with the real world.

Transform

The camera transform (position and orientation) is adjusted by the session based on AR functionality's runtime state. Typically, the session updates the camera's position and orientation using motion tracking data and/or target tracking data to ensure what the user sees matches the real world.

In Unity, the central reference point for all AR tracking is called the session center, and the rules determining this center during session operation are known as the center mode. Camera transform behavior varies under different center modes:

  • In Camera center mode, the camera can move freely.

    Camera mode is rarely used by applications.

  • In other center modes (e.g. FirstTarget), the camera cannot move freely.

    FirstTarget is the mode most AR applications adopt.

Warning

Camera transform scale should always remain (1, 1, 1). Modifying camera scale may cause unexpected behavior.

Projection matrix

The camera projection matrix is updated each frame by the session based on physical camera intrinsics to ensure virtual content aligns properly with real scenes.

Culling settings

Camera culling settings (GL.invertCulling) are adjusted according to session mirroring configurations to ensure correct rendering of virtual content.

When HorizontalFlip corresponds to World for the current camera, GL.invertCulling is set to true. This is the default configuration for front-facing cameras.

AR background video stream

In AR scenes, cameras typically render physical camera video streams as backgrounds to enhance user immersion. The session automatically handles video stream acquisition and rendering while ensuring alignment between video streams and virtual content.

Non-session-controlled cameras

When using headsets, AR Foundation, or FrameSource implementations where IsCameraUnderControl is explicitly set to false, the session doesn't control these camera properties—they're managed by external systems.

Warning

Although the session doesn't control these properties in such cases, third-party systems (like headset SDKs or AR Foundation) manage them. Modifying these properties during app development remains unsupported.

Considerations when copying cameras

When copying parameters from a session camera to another camera, pay special attention to these points:

  • Property acquisition timing: For session-controlled cameras, reference Obtaining session runtime results for correct timing; for non-session-controlled cameras, consult third-party system documentation.
  • FOV, aspect ratio, and projection matrix: Use Camera.projectionMatrix to get and copy the projection matrix. Using Camera.fieldOfView and Camera.aspect alone is insufficient since they are mathematically derived from the projection matrix.

Next steps