Achieving occlusion using the Mega plugin
Occlusion is a key technology for enhancing the immersion of AR virtual-real fusion. This article will guide you on how to achieve occlusion in the xr-frame environment through EasyAR cloud localization and annotations.
Before you begin
- Be able to use Mega Studio in Unity.
- Be able to create and upload annotations using the Unity editor.
- Be able to create content aligned with the real world.
Implementation methods of occlusion
Offline modeling: Use the Unity editor to create 1:1 matching geometric bodies for entities in the real world (such as walls, columns, large equipment) under the Block coordinate system; or obtain optimized models by cropping and decimating the Block dense model.
Runtime alignment: At xr-frame runtime, align the Block coordinate system with the real space through cloud localization and load the corresponding geometric bodies.
Material replacement: Assign special occlusion materials to these geometric bodies.
Visual effect: When the GPU renders other virtual objects, pixels of the occluded parts will be automatically culled due to failing the depth test, thus making virtual objects follow the occlusion logic of real physical space.
How to arrange occlusion with simple geometric bodies
Precisely place cube annotations against the dense model and panoramic images. After placement, the annotations should look like a "wall" or "column".
Modify the annotation name (e.g.,
occlusion_wall), record the ID, and upload the annotation.Use the built-in geometric bodies in the xr-frame mini program to load annotations as occlusion.
In the EMA load callback, use
scene.createElement(xrFrameSystem.XRMesh,{})to create simple geometric bodies and assign theeasyar-occulusionmaterial.Note
Loading, registration, deregistration, and unloading of the
easyar-occulusionmaterial are controlled by the AR Session.
handleEmaResult(ema: easyar.ema.v0_5.Ema) {
let blockHolder: easyar.BlockHolder = session.blockHolder;
ema.blocks.forEach(emaBlock => {
const blockInfo: easyar.BlockInfo = {
id: emaBlock.id
};
// If the Block node does not exist, create a Block node
blockHolder.holdBlock(blockInfo, easyarPlugin.toXRFrame(emaBlock.transform));
});
ema.annotations.forEach(annotation => {
if (annotation.type != mega.EmaV05AnnotationType.Node) {
return;
}
const nodeAnnotation = annotation as easyar.ema.v0_5.Node;
const xrNode: xrfs.XRNode = easyarPlugin.createXRNodeFromNodeAnnotation(nodeAnnotation, blockHolder);
const emaName: string = nodeAnnotation.name;
const geometryStr: string = nodeAnnotation.geometry === "cube" ? "cube" : "sphere";
const assetInfo = AnnotationMetaData[nodeAnnotation.id as keyof typeof AnnotationMetaData];
let model: xrfs.Element;
if (assetInfo) {
// GLTF part
} else {
model = scene.createElement(
xrFrameSystem.XRMesh,
{
// Use the occlusion material registered by the plugin
material: "easyar-occlusion",
// Use xr-frame's built-in geometry, "cube" can also be used directly here
geometry: geometryStr,
name: emaName,
"receive-shadow": "false",
"cast-shadow": "false"
// Do not modify the Scale
}
);
xrNode.addChild(model);
}
})
}
<video src="https://doc-asset.easyar.com/develop/wechat/mega/media/occlusion03.mp4" style="width:480px; max-width:100%; height:auto;" muted playsinline controls></video>
> With occlusion, this panda can dance behind the wall.
How to arrange occlusion for complex geometric bodies
Suitable for scenarios requiring high-precision occlusion such as irregularly shaped equipment or irregular buildings.
You can crop and decimate the Block's dense model to obtain the white model needed for occlusion.
Click the Mega Block node in the Unity scene, and record the BlockID in the Inspector panel.
Select Export under Block in Mega Studio.
Modify export options and export.
1 in the figure is the LOD level; lower levels mean simpler models with fewer faces. Choose 2 for the highest precision; choose 1 or 0 if you can accept reduced precision to decrease face count.
2 in the figure is the texture export option; since we only need a white model for occlusion, textures are not required.
Crop and decimate the exported model in digital content creation software (e.g., Blender), and save it as
Glb.Tip
The example uses Blender's Decimate Modifier.
After cropping and decimation:
Mount the occlusion
Glbfile to a file server to obtain a loadable url.Load the GLTF as occlusion in the xr-frame mini program.
First load the GLTF model for occlusion, then use
scene.createElement(xrFrameSystem.XRGLTF,options)to create the GLTF model.Use
assets.getAsset("material", "easyar-occlusion")to get the material object.Use
model.getComponent(xrFrameSystem.GLTF).meshes.forEach((m: any) => {m.setData({ neverCull: true, material: occlusionMaterial });}to modify the material of the GLTF model.Note
Loading, registration, deregistration, and unloading of the
easyar-occulusionmaterial are controlled by the AR Session.
const sampleAssets = {
occlusion1: {
assetId: "occlusion1",
type: "gltf",
src: "url/occlusion1.glb",
options: {}
}
}
async loadAsset() {
if (!scene) {console.error("Empty scene"); return;}
try {
await scene.assets.loadAsset(sampleAssets.occlusion1);
} catch (err) {
console.error(`Failed to load assets: ${err.message}`);
}
},
addOcclusion() {
model = scene.createElement(
xrFrameSystem.XRGLTF,
{
"model": assetInfo.assetId,
"anim-autoplay": assetInfo.animation ? assetInfo.animation : "",
"scale": assetInfo.scale ? assetInfo.scale : "1 1 1",
name: "tree"
}
);
const blockID = "aaaa1234-bbbb-cccc-dddd-eeeeee123456" //Fill in the Block ID here
if (!blockHolder.getBlockById(blockParent.id)) {
// If no existing Block node exists, create one
blockHolder.holdBlock({
id: blockID
})
}
// Get the Block node in the xr-frame scene
let blockElement = blockHolder.getBlockById(blockParent.id).el;
// Mount the clipped occlusion model as a child node under the Block node
blockElement.addChild(model);
/**
* Due to differences in GLTF loader behavior, to ensure the model's orientation on xr-frame exactly matches Unity's rendering result
* it is sometimes necessary to rotate the loaded model 180 degrees around the Y-axis in place
*/
let modelTransform = model.getComponent(xrFrameSystem.Transform);
let currentRotation = modelTransform.quaternion.clone();
let targetRotation = currentRotation.multiply(new xrFrameSystem.Quaternion().setValue(0, 1, 0, 0));
modelTransform.quaternion.set(targetRotation);
//Note: Material modification must occur after modifying the Transform
if (assetInfo.assetId == 'occlusion1') {
//Get the occlusion material provided by the mega plugin
let occlusionMaterial = scene.assets.getAsset("material", "easyar-occlusion");
//Modify the occlusion material
model.getComponent(xrFrameSystem.GLTF).meshes.forEach((m: any) => {
m.setData({ neverCull: true, material: occlusionMaterial });
});
}
}
> [!NOTE]
> Using the cropped Mega Block dense model for occlusion does not require annotation to synchronize spatial position. This is because in digital content creation software (e.g., Blender), decimation and cropping can be performed without changing the coordinate system definition.
>
> To precisely place your own GLTF occlusion model, refer to [How to place a spatially aligned occlusion model](./sample.md#wechat-mega-sample-precise-occulusion-model)
The final device running effect is shown in the video at the top of the article.
Expected occlusion effect
The occlusion effect on the xr-frame mini program is mainly affected by:
- Accuracy of localization and tracking itself
- Accuracy of model placement
- Precision of the model itself (if not a simple geometric body)
Misalignment of several centimeters during localization drift is normal.
Occlusion models with too many faces can easily affect performance. It is recommended to use them only in necessary areas and to use simple geometric bodies for occlusion whenever possible.