Table of Contents

Play transparent video on an xr-frame WeChat Mini Program

Before you start

  • Prepare the transparent video to be played: upload the transparent video to a file server and obtain the URL used for loading it in xr-frame.

  • Because transparent video playback depends on replacing textures in the scene, the location where the transparent video will be played needs to be annotated in advance. You need to be able to create and upload annotations using the Unity Editor.

What is transparent video

Transparent video is a technical solution for achieving a transparent background effect in video encoding formats that do not natively support alpha channels, such as H.264/AVC and H.265/HEVC.

This solution splits the color information (RGB) and transparency information (Alpha) of the video image, and stitches them into the same frame image according to a specific spatial layout, generating a normal video file with a black-and-white mask. On the playback side, the graphics rendering pipeline samples in real time and composites the two parts to restore dynamic images with a transparent background.

According to how the color area and Alpha area are stitched, there are two main formats:

  1. Side-by-Side (SBS)

    Side-by-Side is a format that stitches the RGB color frame and the Alpha mask frame side by side in the horizontal direction. Usually, the left side is the color area and the right side is the corresponding grayscale Alpha area.

  2. Top-by-Bottom (TBB)

    Top-by-Bottom is a format that stacks and stitches the RGB color frame and the Alpha mask frame in the vertical direction. Usually, the upper half is the color area and the lower half is the corresponding grayscale Alpha area.

Play transparent video at annotated positions in an xr-frame Mini Program

First load a video resource of type video-texture.

async loadAsset() {
    const videoTexture = {
        assetId: "fireball",
        type: "video-texture",
        // 视频资源 URL
        src: "url/video-resource.mp4",
        options: {
            autoPlay: true,
            loop: true,
        }
    };
    try {
        // 加载 video-texture 类型资源
        await scene.assets.loadAsset(videoTexture);
    } catch (err) {
        console.error(`Failed to load video texture: ${err.message}`);
    }
}

In the callback loaded by EMA, use scene.createElement(xrFrameSystem.XRMesh,{}) to create a simple geometry, assign the easyar-video-tsbs material, and change uniform to u_baseColorMap:video-{$assetId}.

Note

Loading, registration, deregistration, and unloading of the easyar-video-tsbs and easyar-video-ttbb materials are controlled by AR Session.

handleEmaResult(ema: easyar.ema.v0_5.Ema) {
    const blockHolder: easyar.BlockHolder = session.blockHolder;
    ema.blocks.forEach(emaBlock => {
        const blockInfo: easyar.BlockInfo = {
            id: emaBlock.id
        };
        // 若 Block 节点不存在，创建 Block 节点
        blockHolder.holdBlock(blockInfo, easyarPlugin.toXRFrame(emaBlock.transform));
    });
    ema.annotations.forEach(annotation => {
        if (annotation.type !== mega.EmaV05AnnotationType.Node) {
            return;
        }
        const nodeAnnotation = annotation as easyar.ema.v0_5.Node;
        const xrNode: xrfs.XRNode = easyarPlugin.createXRNodeFromNodeAnnotation(nodeAnnotation, blockHolder);
        const assetInfo = AnnotationMetaData[nodeAnnotation.id as keyof typeof AnnotationMetaData];
        let model: xrfs.Element;

        if (assetInfo) {
            // GLTF 部分
        } else {
            // 利用内置 Mesh 创建用于渲染的几何体
            model = scene.createElement(xrFrameSystem.XRMesh, {
                geometry: "cube",
                material: "easyar-video-tsbs",
                uniforms: "u_baseColorMap:video-fireball",
            });
            xrNode.addChild(model);
        }
    });
}
Note

When using video-texture, if the console shows the warning wx.createVideoDecoder with type: 'wemedia' is deprecated, ignore it.

It has been confirmed with the WeChat official team that this warning does not affect usage.

Next steps