When I was working on Video Ops, I faced numerous challenges, some of which were completely new to me. One such challenge was figuring out how to handle thumbnails - specifically, how to store and serve them effectively.

Suboptimal solutions

One potential solution would be to store one image for every few seconds of video, but this presents a problem if videos have varying amounts of images. Additionally, for shorter videos, the number of thumbnails generated would be insufficient. Another option would be to store 1 image per 1% of the video duration, but requesting 100 images over the network is far from ideal. So after exploring popular video streaming services, I found that YouTube sends images in a collage format, but in different amounts. To simplify the process, I opted for the second option, combined with the approach used on YouTube. I store 100 images per video in a collage and retrieve them on the front end using Javascript. This results in seamless and consistent thumbnail scrolling across all videos.

Generating images from video

Fortunately, getting images from a video is not too difficult with FFmpeg. Here's a function that takes care of it using fluent-ffmpeg:

function ffmpegScrn(input: string, duration: number) {
   const outputFileName = `${input.split(".")[0]}.webp`;
   const frameInterval = duration / 100;
   const outputStream = bucket_prod.file(outputFileName).createWriteStream();
   return new Promise((resolve, reject) => {
      ffmpeg(input) 
         .setFfmpegPath(ffmpegPath) //path to binary
         .outputOptions([
            `-vf fps=1/${frameInterval},scale=128:72:force_original_aspect_ratio=decrease,pad=128:72:-1:-1:color=black,tile=10x10`,
            "-frames:v 1",
            "-q:v 50",
         ])
         .output(outputFileName)
         .on("end", async () => {
            createReadStream(outputFileName)
               .pipe(outputStream)
               .on("finish", () => {
                   //delete output after upload
                   fs.unlink(outputFileName);
                   resolve(outputFileName);
               });
         })
         .on("error", (err) => {
            reject(err.message);
         })
         .run();
   });
}

The next step is to split HTMLCanvas into 100 parts and save them to the variable. Since the image collage is 1280x720, it is quite easy to divide it into a 10x10 matrix.

// ....
// loop through each thumbnail in the collage and extract it
const thumbnailWidth = 128;
const thumbnailHeight = 72;
const numThumbnails = 100; // set the number of thumbnails in the collage
const thumbnails = [];
// tmp canvas to put ImageData derived from main canvas
const tmpCanvas = document.createElement("canvas");
const tmpContext = tmpCanvas.getContext("2d");
tmpCanvas.width = thumbnailWidth;
tmpCanvas.height = thumbnailHeight;
for (let i = 0; i < numThumbnails; i++) {
   // calculate the position of the current thumbnail in the collage
   const x = (i % 10) * thumbnailWidth;
   const y = Math.floor(i / 10) * thumbnailHeight;
   // extract the current thumbnail from the canvas
   const thumbnail = context.getImageData( x, y, thumbnailWidth, thumbnailHeight );
   tmpContext!.putImageData(thumbnail, 0, 0);
   const url = await getCanvasBlobUrl(tmpCanvas);
   thumbnails.push(url);
}
//remove source
tmpCanvas.remove();
canvas.remove();
} 
// ...

You may be curious about the process of converting canvas images into URLs using getCanvasBlobUrl. This process is accomplished through the use of the toBlob function.

function getCanvasBlobUrl(canvas: HTMLCanvasElement): Promise<string> {
   return new Promise((resolve) => {
      canvas.toBlob((blob) => {
         const url = URL.createObjectURL(blob!);
         resolve(url);
      });
   });
}

After the process, we obtain 100 blob URLs that can be used for the video player. However, it is crucial to keep in mind that these object URLs should be deleted once they are no longer needed.

function deleteImages(thumbnails: string[]) {
   for (let i = 0; i < thumbnails.length; i++) {
      URL.revokeObjectURL(thumbnails[i]);
   }
}

You can find the complete source code in the GitHub repo.