5.0 Stencil Buffer

The stencil buffer is closely related to the depth buffer. When stencil testing is enabled, the rendering pipeline performs a visibility test based on the contents of the stencil buffer. The exact formula for visibility testing can be confirmed through configuration. Typically, this process involves two passes: the first pass populates the stencil buffer, while the second pass handles the actual rendering. Only the portions of the scene that pass the visibility test will be rendered.

The stencil buffer is particularly useful for creating certain effects in games, such as mirrors or portals. For example, to render a mirror effect, you first draw the mirror's shape into the stencil buffer. Then, with stencil testing enabled, you render the scene from the perspective of the mirror. Many intriguing illusion games, such as the Non-Euclidean Worlds Engine, leverage advanced stencil buffer techniques.

As mentioned earlier, using the stencil buffer typically involves two passes. The first pass populates the stencil buffer, which may seem counterintuitive because you cannot directly write to the stencil buffer in the fragment shader. Instead, the stencil buffer is populated as a byproduct of rendering, similar to the depth buffer. The values written to the stencil buffer are preconfigured.

In this tutorial, we will create a simple portal effect akin to the Non-Euclidean Worlds Engine project. The concept is straightforward: we first render a virtual gateway, which is a simple plane, to populate the stencil buffer. Next, we render the scene with stencil visibility testing enabled. Finally, we render the frame of the gateway.

Gateway 3D Model

Let's first examine the shader code used to populate the stencil buffer. This shader is designed to be as simple as possible. It calculates the clip coordinates and renders the object as transparent pixels. The goal here is not to render anything visible but to generate stencil values. Similar to the depth buffer, stencil buffer writing is enabled so that whenever a fragment is produced, a stencil value is also cached into the stencil buffer.

@group(0) @binding(0)
var<uniform> modelView: mat4x4<f32>;
@group(0) @binding(1)
var<uniform> projection: mat4x4<f32>;

struct VertexOutput {
    @builtin(position) clip_position: vec4<f32>,
};

@vertex
fn vs_main(
    @location(0) inPos: vec3<f32>
) -> VertexOutput {
    var out: VertexOutput;
    var wldLoc:vec4<f32> = modelView * vec4<f32>(inPos, 1.0);
    out.clip_position = projection * wldLoc;
    return out;
}

@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
    return vec4<f32>( 0.0,0.0,0.0,0.0);
}

Next, let's explore how stencil writing is enabled when creating the pipeline:

const pipelineDesc = {
    layout,
    vertex: {
        module: shaderModule,
        entryPoint: 'vs_main',
        buffers: [positionBufferLayoutDesc, normalBufferLayoutDesc]
    },
    fragment: {
        module: shaderModule,
        entryPoint: 'fs_main',
        targets: [colorState]
    },
    primitive: {
        topology: 'triangle-strip',
        frontFace: 'ccw',
        cullMode: 'none'
    },
    depthStencil: {
        depthWriteEnabled: true,
        depthCompare: 'less',
        format: 'depth24plus-stencil8',
        stencilFront: {
            compare: "less",
            passOp: "keep",
        },
        stencilBack: {
            compare: "less",
            passOp: "keep",
        }
    }
};

Notice that the format depth24plus-stencil8 specifies 24 bits for depth and 8 bits for the stencil per pixel. The stencilFront and stencilBack fields, which we hadn't configured before, define how visibility tests are performed for front-facing and back-facing fragments, respectively. Each field requires a comparison method for the visibility test and operations to update the stencil buffer. In this example, the comparison method is set to always, meaning the visibility test will always pass.

There are three types of operations to define: passOp, failOp, and depthFailOp. These determine how to update the stencil buffer when the stencil test passes, fails, or when the depth test fails, respectively. The default operation is keep, which retains the existing stencil buffer value. In this example, we use the replace operation, which replaces the existing value with a new value that we will specify later.

You might wonder why we have depthFailOp in the stencil test configuration. What does the stencil test have to do with the depth test? The reason is that the stencil test occurs before the depth test during rendering. In some cases, a fragment might pass the stencil test but fail the depth test. The depthFailOp operation allows you to specify how to handle such situations.

In essence, the pipeline configuration we discussed renders the portal geometry into the stencil buffer. This means that all pixels covered by the portal are assigned the specified stencil value.

Here's how we set the stencil reference value:

const passEncoder = commandEncoder.beginRenderPass(renderPassDesc);
passEncoder.setViewport(0, 0, canvas.width, canvas.height, 0, 1);
passEncoder.setStencilReference(0xFF);
stencil.encode(passEncoder);
passEncoder.end();

In this code snippet, the setStencilReference function is used to set the stencil reference value to 0xFF. Given that we only have 8 bits for each pixel in the stencil buffer, this reference value will ensure that only pixels matching 0xFF will pass the stencil test. Pixels not covered by the portal are set to 0.

In the second rendering pass, we will draw the actual scene, which includes a teapot and a ground plane. However, we want the scene to be visible only through the portal. To achieve this, we perform a visibility test using the stencil buffer populated during the first pass.

const pipelineDesc = {
    layout,
    vertex: {
        module: shaderModule,
        entryPoint: 'vs_main',
        buffers: [positionBufferLayoutDesc, normalBufferLayoutDesc]
    },
    fragment: {
        module: shaderModule,
        entryPoint: 'fs_main',
        targets: [colorState]
    },
    primitive: {
        topology: 'triangle-list',
        frontFace: 'ccw',
        cullMode: 'none'
    },
    depthStencil: {
        depthWriteEnabled: true,
        depthCompare: 'less',
        format: 'depth24plus-stencil8',
        stencilFront: {
            compare: "less",
            passOp: "keep",
        },
        stencilBack: {
            compare: "less",
            passOp: "keep",
        }
    }
};

In this updated pipeline description, the compare function is set to less, meaning that the stencil test will pass if the stencil reference value is less than the value currently stored in the stencil buffer. The passOp is set to keep, which means that the stencil buffer will remain unchanged by this rendering pass. This configuration ensures that the stencil buffer’s values are preserved while we perform the rendering operation.

const pipelineDesc = {
    layout,
    vertex: {
        module: shaderModule,
        entryPoint: 'vs_main',
        buffers: [positionBufferLayoutDesc]
    },
    fragment: {
        module: shaderModule,
        entryPoint: 'fs_main',
        targets: [colorState]
    },
    primitive: {
        topology: 'triangle-list',
        frontFace: 'ccw',
        cullMode: 'none'
    },
    depthStencil: {
        depthWriteEnabled: true,
        depthCompare: 'less',
        format: 'depth24plus-stencil8',
        stencilFront: {
            compare: "always",
            passOp: "replace",
        },
        stencilBack: {
            compare: "always",
            passOp: "replace",
        }
    }
};

const passEncoder2 = commandEncoder.beginRenderPass(renderPassDesc2);
passEncoder2.setViewport(0, 0, canvas.width, canvas.height, 0, 1);
passEncoder2.setStencilReference(0x0);
plane.encode(passEncoder2);
teapot.encode(passEncoder2);
passEncoder2.end();

Again, when generating the command buffer, we need to set the stencil reference value. This time, we set it to zero. Since zero is less than 0xFF, the stencil test will pass wherever the value in the stencil buffer is 0xFF. For areas where the stencil buffer value is zero (equal to our reference value), the stencil test will fail.

Finally, in the last rendering pass, we want to render the portal's frame. In this case, we do not want any stencil testing, so we set the comparison function to always. Additionally, we want to ensure that this pass does not alter the stencil buffer, so we use the keep operation to maintain the existing stencil values.

const pipelineDesc = {
    layout,
    vertex: {
        module: shaderModule,
        entryPoint: 'vs_main',
        buffers: [positionBufferLayoutDesc, normalBufferLayoutDesc]
    },
    fragment: {
        module: shaderModule,
        entryPoint: 'fs_main',
        targets: [colorState]
    },
    primitive: {
        topology: 'triangle-list',
        frontFace: 'ccw',
        cullMode: 'none'
    },
    depthStencil: {
        depthWriteEnabled: true,
        depthCompare: 'less',
        format: 'depth24plus-stencil8',
        stencilFront: {
            compare: "always",
            passOp: "keep",
        },
        stencilBack: {
            compare: "always",
            passOp: "keep",
        }
    }
};

In this final rendering pass, the stencil reference value is irrelevant because we are not performing a stencil test and will not modify the stencil buffer.

Screenshot of the Demo