Overcoming IndexedDB Limits by Streaming Videos

Since we last left off, I've been hard at work on my game and working towards making it playable for others. Last time I wrote a post, I had just gotten the realtime scoring working and the game was playable!

There was one glaring problem though: I ran out of IndexedDB storage.

What does this mean, and how did I overcome it? Let's dive in!

IndexedDB Limits

In an earlier blog post, I talked about how to build more than a YouTube video downloader. The reason behind building this video downloader server was to get around the problem where TensorFlow.js can't analyze YouTube videos because we don't have access to the <video> element when it is embedded in an iframe.

This video downloader approach does the following:

1. It takes a YouTube video link and uses youtube-dl to download it to a server running in the cloud
2. It uploads that video to a cloud storage solution
3. It sends the link to download the video from the cloud storage
4. Once you receive the link to download the video, your browser "silently" downloads the video file to your IndexedDB storage as a blob

It works pretty well, but it's not scalable. After creating about 10 or so projects with varying file sizes for the video downloads (usually between 30mb-200mb for a 1080p video), I hit an error telling me that my IndexedDB storage was full.

I thought I had more space for video downloads! After looking up the IndexedDB storage limits, it turns out that I can only use a maximum of 2 GB of IndexedDB storage for a website.

I had to rethink my approach for serving videos to players. How could I use YouTube videos and still be able to run TensorFlow.js on them on my website?

Streaming Videos From Cloud Storage

What if I controlled the streaming of the videos instead of YouTube? I already have all the videos being downloaded and uploaded to cloud storage - why not just embed the videos without downloading them?

This way, I still have access to the <video> element, but I don't have to download them to the IndexedDB storage.

After a few days of coding this new solution, I had it working! It worked pretty much the same as before, except that loading new projects was slightly faster since you didn't have to download the entire video at once.

The new project creation flow looks a little something like this:

1. Enter project title and YouTube link
2. Wait for the server to download the YouTube video and upload it to cloud storage
3. Server sends client the link, and the client uses that link to stream the video as they play it

The project editor experience was nearly the same as before when I wasn't streaming since the streaming approach was so quick to load in most cases. And the TensorFlow.js analysis still worked perfectly, so we've successfully solved the problem.

But we can do better!

The Power of Browser Extensions

Browser extensions, such as ad-blocking ones or password managers, are incredibly powerful and useful tools. They can often greatly improve your day to day experience when browsing the web.

When I say that they're incredibly powerful though, I mean it! They can read and modify almost anything on any webpage if you allow certain permissions. That's a little scary!

And because they're so powerful, it turns out that they can also run the MoveNet model from TensorFlow.js on any website. I pondered this for a little while and developed a plan...

YouTube Videos DO Mix with TensorFlow.js

In an earlier blog post, I detailed how it's not possible to run TensorFlow.js' MoveNet on an embedded YouTube video because the <video> element is tucked away in an iframe, which is inaccessible to our website's code.

The interesting thing about browser extensions is that they have access to all elements displayed in your browser, whether they are in an iframe or not.

With a little help from a browser extension, we can run TensorFlow.js on any embedded YouTube video.

The New Project Creation Flow

With this new, extra shiny streaming approach in mind (YouTube is doing all the work now, yay!), let's go over the project creation flow again:

1. Enter project title and YouTube link
2. Instantly enter the project without waiting for anything to download

That's...incredibly streamlined and simple, right? And now onto the fun bit: how do we actually analyze the video to get the pose and keypoint data back to our website and into our project?

Data Flow Between Our Website and Our Extension

Communicating back and forth between our website and our extension is an interesting problem to solve! Let's dive into it.

Browser extensions usually have at least three different parts: the browser popup, the content script, and the background script.

The Browser Popup

The browser popup is the interface that pops up when you click the extension's icon in the top right corner after installing it. Usually, this view has configurable settings and information about the extension.

We won't be using this for anything important.

The Content Script

When a browser extension is configured to run in a specific domain, such as any YouTube page, it injects its own content script into the webpage. This content script is a normal JavaScript file that can do anything and everything on the page, which can make it a little spooky.

We'll be using two types of content scripts for our purposes: a main and an iframe content script.

The Main Content Script

The main content script is a simple script with the sole purpose of passing messages between different mediums in the browser. This script is injected into our game's webpage.

There are two types of messages:

• Messages from the background script (through browser.runtime.onMessage.addListener)
• Messages from the webpage itself (through top.addEventListener)

The Iframe Content Script

By default, content scripts will be inject themselves into iframes on the webpage. There's a special config option in the extension manifest to enable it, like so: "all_frames": true. This causes this specific content script to only inject itself into iframes, such as YouTube embeds, and not the main webpage.

This script is a lot more complicated than the main content script, since it needs to handle all logic involving TensorFlow.js. Specifically, it listens for various types of events, such as those to start and stop video analysis. It uses browser.runtime.onMessage.addListener to listen for events from the background script, and it sends messages back to the background script using browser.runtime.sendMessage.

The Background Script

The background script is a special type of script because it runs in the background of your browser and isn't injected into any webpage. Because of this, it doesn't know anything about what's on a specific webpage.

For our use case, we'll be using the background script to pass messages between content scripts, since content scripts on webpages and iframes cannot talk to each other. It also has a couple other purposes that we won't dive into here.

To pass messages along to a specific tab, we can use these functions to find the active tab and send a message to it:

function sendMessageToTabs(tabs, data) {
    for (let tab of tabs) {
        browser.tabs
            .sendMessage(tab.id, data)
            .catch(onError);
    }
}

function sendMessageToThisTab(data) {
    browser.tabs
        .query({
            currentWindow: true,
            active: true,
        })
        .then((tabs) => {
            sendMessageToTabs(tabs, data);
        })
        .catch(onError);
}

When listening for messages from the content scripts, we can use browser.runtime.onMessage.addListener.

Data Flow Summary

The data flow between all these parts looks like this:

It's a little bit convoluted just to communicate between the game website and the YouTube iframe, but that's the best solution I was able to come up with! And it actually works really well - you can pass almost any sort of data between the website and the iframe.

With this all figured out though, we now have a working streaming solution where we're able to analyze YouTube videos using TensorFlow.js!

Our game website sends a message asking to start analysis, and this message eventually gets to the content script in the iframe of the YouTube video. This content script starts the analysis and sends back the results once it's done, and these results are used on the normal website.

Will Everyone Need to Use the Browser Extension?

Nope! That's a really nice part about this approach as well - you only need the browser extension if you want to analyze YouTube videos to create new dance charts from scratch.

If you just want to play existing charts using your webcam, you won't need to install anything to be able to play. Just jump right in!

Next Steps

What's next for our dance game? It's time to make the Play screen where players are able to browse various songs and play them! They'll be able to see published dance charts by others players as well as high scores on those charts.

Once these steps are done, I'll be able to start sending out beta invitations for a small group of people to begin testing the game!