2.3 Video Rendering
We've learned how to load and render images as textures. Similarly, we can load a video as a texture, which is particularly useful for creating web-based video processing applications.
A video can be viewed as a series of images, so it's not fundamentally different from image textures. The key challenge is updating the images efficiently. In this tutorial, we'll explore two methods: manual frame updates and automatic updates via external textures.
Manual Frame Update
Launch Playground - 2_03_1_video_1First, we need a function that, given a URL to a video file, installs a hidden video tag in the document. We then monitor the video playback.
function setupVideo(url) {
const video = document.createElement('video');
let playing = false;
let timeupdate = false;
video.playsInline = true;
video.muted = true;
video.loop = true;
// Waiting for these 2 events ensures
// there is data in the video
video.addEventListener('playing', () => {
playing = true;
checkReady();
}, true);
video.addEventListener('timeupdate', () => {
timeupdate = true;
checkReady();
}, true);
video.src = url;
video.play();
function checkReady() {
if (playing && timeupdate) {
copyVideo = true;
}
}
return video;
}
videoTag = setupVideo('../data/Firefox.mp4');
(async () => {
return new Promise(resolve => {
let timer = setInterval(() => {
if (copyVideo) {
clearInterval(timer);
resolve();
}
}, 300);
})
})().then(() => {
webgpu();
});
We need to watch both the playing and timeupdate events. Only when both have occurred can we be sure the video has started. If we don’t wait for the video to start, there might be no video data available, and the video size might be unassigned. Therefore, we must wait for the video to start before proceeding.
const textureDescriptor = {
size: { width: videoTag.videoWidth, height: videoTag.videoHeight },
format: 'rgba8unorm',
usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT
};
const texture = device.createTexture(textureDescriptor);
videoTag.ontimeupdate = async (event) => {
let imageData = await createImageBitmap(videoTag);
//console.log(imageData)
device.queue.copyExternalImageToTexture({ source: imageData }, { texture }, textureDescriptor.size);
};
The size of the texture map matches that of the video tag. Note that we need to request RENDER_ATTACHMENT usage for this texture map, as it is required by the copyExternalImageToTexture function call. Finally, we create an event listener for the timeupdate event. In this event handler, we create a bitmap using the createImageBitmap function and load its content into the texture map via copyExternalImageToTexture.
The rest of the rendering code remains the same as before, so we will skip those details.
Importing External Textures
Launch Playground - 2_03_2_video_2In the previous example, we manually copied data whenever a video frame updated. An alternative approach is to import an external texture from the video tag. This method allows us to avoid creating a texture map ourselves, but we need to be mindful of the lifetime of the imported texture. The texture can expire if the video switches to a new frame or when the VideoFrame's close() function is called.
function render() {
requestAnimationFrame(render);
if (copyVideo) {
const externalTexture = device.importExternalTexture({ source: videoTag });
let uniformBindGroup = device.createBindGroup({
layout: uniformBindGroupLayout,
entries: [
{
binding: 0,
resource: {
buffer: translateMatrixUniformBuffer
}
},
{
binding: 1,
resource: {
buffer: projectionMatrixUniformBuffer
}
},
{
binding: 2,
resource: externalTexture
},
{
binding: 3,
resource:
sampler
}
]
});
commandEncoder = device.createCommandEncoder();
passEncoder = commandEncoder.beginRenderPass(renderPassDesc);
passEncoder.setViewport(0, 0, canvas.width, canvas.height, 0, 1);
passEncoder.setPipeline(pipeline);
passEncoder.setBindGroup(0, uniformBindGroup);
passEncoder.setVertexBuffer(0, positionBuffer);
passEncoder.setVertexBuffer(1, texCoordsBuffer);
passEncoder.draw(4, 1);
passEncoder.end();
device.queue.submit([commandEncoder.finish()]);
}
}
requestAnimationFrame(render);
}
To avoid texture expiration, it's crucial to import and use the texture immediately within the same render function call. This necessity is why we create a bind group for each rendering frame to incorporate the new external texture map.