3.3 Saving Images and Videos

In this tutorial, we'll explore how to save rendered content as images and videos. This capability is useful for applications involving video and image editing.

Saving images is straightforward, thanks to the canvas element’s built-in method for exporting content to a blob. In the demo, I’ve added the image-saving logic at the end of the rendering function to ensure that when a user clicks "Save Image," the content is up-to-date. Thus, we first render the scene again before saving, and only save the image once rendering is complete.

let takeScreenshot = false;

screenshotButton.onclick = () => {
    takeScreenshot = true;
}

if (takeScreenshot) {
    takeScreenshot = false;
    canvas.toBlob((blob) => {
        saveBlob(blob, `screencapture-${canvas.width}x${canvas.height}.png`);
    });
}

However, a blob is not directly visible as an image; it needs to be saved as a file. For this, we use a helper function:

const saveBlob = (function () {
    const a = document.createElement('a');
    document.body.appendChild(a);
    a.style.display = 'none';
    return function saveData(blob, fileName) {
        const url = window.URL.createObjectURL(blob);
        a.href = url;
        a.download = fileName;
        a.click();
        a.parentNode.removeChild(a);
    };
}());

This function works by creating a hidden link element, encoding the blob as an object URL using window.URL.createObjectURL(blob), and programmatically clicking the link to trigger the download before removing it from the DOM.

The above example demonstrates how to save the canvas as an image. Sometimes, it’s also useful to dump textures or intermediate buffers created during a render pass for debugging purposes. We will cover this in later chapters.

Saving videos is more complex. In the demo, we use a new API called WebCodecs, which allows us to encode video frames directly in the browser.

Most APIs in the computing realm are organized hierarchically, ranging from low-level APIs that are difficult to use but highly flexible to high-level APIs that are easier to use but may be limited to specific use cases.

Web standards differ in this regard. Many web APIs are designed to perform one specific function. For example, before WebCodecs, we had the <video> tag for playing videos and WebRTC for video conferencing, but neither provided a low-level video API for encoding frames.

This changed with the advent of cloud gaming. As cloud gaming introduced higher demands for video stream control to ensure low-latency gameplay, Google introduced WebCodecs to offer low-level video APIs.

We will use the same button to start and stop video recording.

recordButton.onclick = async () => {
    if (encoder === null) {
        const options = {
            suggestedName: 'video.webm',
            types: [
                {
                    description: 'Webm video',
                    accept: {
                        'video/webm': ['.webm']
                    }
                }
            ],
        };

        let fileHandle = await window.showSaveFilePicker(options);

        fileWritableStream = await fileHandle.createWritable();
        // This WebMWriter thing comes from the third-party library
        webmWriter = new WebMWriter({
            fileWriter: fileWritableStream,
            codec: 'VP9',
            width: canvas.width,
            height: canvas.height
        });
        encoder = new VideoEncoder({
            output: chunk => webmWriter.addFrame(chunk),
            error: e => console.error(e)
        });
        // Configure to your liking
        encoder.configure({
            codec: "vp09.00.10.08",
            width: canvas.width,
            height: canvas.height,
            bitrate: 2_000_000,
            latencyMode: 'realtime',
            framerate: 25
        });
        frameCount = 0;
        beginTime = window.performance.now();
        recordButton.innerText = 'Stop';
    } else {
        encoder.flush();
        await webmWriter.complete();
        fileWritableStream.close();

        webmWriter = null;
        fileWritableStream = null;
        encoder = null;
        recordButton.innerText = 'Record';
    }
}

When the encoder is not yet created, we initiate recording. The first step is to show a file picker dialog for the user to specify a filename for the video:

const options = {
    suggestedName: 'video.webm',
    types: [
        {
            description: 'Webm video',
            accept: {
                'video/webm': ['.webm']
            }
        }
    ],
};

let fileHandle = await window.showSaveFilePicker(options);

fileWritableStream = await fileHandle.createWritable();

For the file format, we are using WebM, a free video format developed by Google. If you are unfamiliar with video formats, here is a brief overview: most video files are actually container formats that can hold multiple media streams or tracks. For example, a movie might include a video track and an audio track, each encoded with different codecs.

When using WebCodecs for encoding, we can only encode individual frames. These frames need to be assembled into a track and then placed into a container format to be playable. Unfortunately, muxing (the process of assembling a container format) is not part of the WebCodecs standard. Therefore, we use a third-party library called webm-writer2.js to handle this task.

Once we have the writable stream, we create the muxing object:

webmWriter = new WebMWriter({
    fileWriter: fileWritableStream,
    codec: 'VP9',
    width: canvas.width,
    height: canvas.height
});

We use the VP9 codec, a common choice for WebM, and set the width and height to match the canvas size.

encoder = new VideoEncoder({
    output: chunk => webmWriter.addFrame(chunk),
    error: e => console.error(e)
});
// Configure to your liking
encoder.configure({
    codec: "vp09.00.10.08",
    width: canvas.width,
    height: canvas.height,
    bitrate: 2_000_000,
    latencyMode: 'realtime',
    framerate: 25
});

Finally, we create and configure the encoder. The encoder uses two callbacks: the output callback, which is triggered when a frame is encoded, and the error callback, which is called if an error occurs. We pass the data chunk to the webmWriter to assemble the video file.

To configure the encoder, we specify VP9 as the codec, along with parameters such as bitrate and framerate. We also initialize the frameCount to zero and capture a high-precision timestamp. The frameCount helps determine when to encode keyframes versus intra-frames. Keyframes typically use spatial compression, while intra-frames use both spatial and temporal compression. Intra-frames have a higher compression rate but cannot be decompressed on their own; they require the previous keyframe and all intra-frames between them to be decoded. Due to the accumulation of compression errors, we periodically encode keyframes, which can be decoded independently but are larger in size compared to intra-frames.

Encoding a frame also requires a timestamp, so we record the time when video recording begins.

frameCount = 0;
beginTime = window.performance.now();

Similar to the image dumping code, we also place the video encoding code after a frame is rendered. If the encoder has been initialized, we first capture the current time. We then create a VideoFrame using the canvas and the elapsed time since the recording began. We call the encode function, and every 10 frames, we encode a keyframe.

async function render() {

    if (encoder !== null) {
        let currentTime = window.performance.now();
        const frame = new VideoFrame(canvas, { timestamp: (currentTime - beginTime) * 1000 });
        encoder.encode(frame, { keyFrame: frameCount % 10 == 0 });
        frame.close();
        frameCount++;
    }

    requestAnimationFrame(render);
}

When the user clicks the record button again, we stop recording and save the video file:

encoder.flush();
await webmWriter.complete();
fileWritableStream.close();

webmWriter = null;
fileWritableStream = null;
encoder = null;
recordButton.innerText = 'Record';

Here, we first flush the encoder to complete encoding any pending frames. Next, we close the webmWriter and the file stream. The saved file should now be playable with a video player.