Difference between revisions of "Assets Distribution"

Latest revision as of 09:38, 17 August 2016

First, read this: Mesh Computing and Tootsville

So, how does this “Mesh Computing” thing work for distribution of media assets, like music, sound effects, and graphics pieces?

We have one, central repository named jumbo.tootsville.adventuring.click that's responsible for storing the canonical, original forms of all of our assets. In the simplest case, your web browser reads the web page that you've just visited, and goes through looking for links to these media assets. It then connects up to the asset server and asks for a copy of that one resource. This is all well and good for most things.

Bundling[edit]

Each time you connect to the asset server, though, there's a “time” cost. So, the next stage of optimizing a web site might be to make a “sprite sheet” or other bundle of assets. For example, we could take a number of related 2D art pieces and put them together into one larger file. The amount of effort — both time, and network bandwidth — required to grab that larger piece will be lower, for two reasons.

On the one hand, it's a bit like ordering a dinner á la carte, versus ordering a pre-arranged meal. Think of fast food: Saying “I'll have the #1 combo” is quicker to say than “I want a sandwich, and fries, and a cola.” Over HTTP, it's actually even worse than all that: it's more like walking through the line at the restaurant three times, once to get a sandwich, once to get fries, and once to get a cola.

The other advantage of bundling the assets is that we compress almost everything on the Internet. Compression systems work better, the larger the amount of material they're given. So, taking 16 different icons and compressing them individually, versus combining them into one file and compressing it as a whole, the single file will compress better.

Of course, this is only a “win” when you actually need most of the 16 icons in that bundle. Sites like Gmail use this technique because they “know” most users will need almost all of the icons.

In a game context, we have to cheat a bit. We create bundles based upon actual demand from users. It could be that one bundle is created with assets that occur together in Toot Square, because anyone walking into Toot Square would be expected to use most of those assets.

The assets are each “rated” for what their “proximity” is to other assets. This means that we may have a few relatively-cohesive groups of assets, and a few assets that are rarely accessed and may even end up alone.

As these bundles are created, we stage them for distribution. Alongside this, the Asset Catalog is updated.

The Catalog[edit]

The Catalog is produced based on this bundling logic. Over time, as the pattern of resource needs changes, players' computers can “vote” to change the bundling.

The Catalog is date-stamped, and from time to time, a new version will be created, along with newly-rebundled asset packs. Users will eventually start fetching assets from the new Catalog, but they could keep around asset packs that they've previously fetched from an older version as well. No need to re-download something that they already have cached.

The details of the bundling and catalog system should be documented, but are very much in flux now.

WebTorrent[edit]

In order to reduce the workload on Jumbo, we try to avoid fetching assets directly from the host. This is accomplished via Torrent architecture.

The WebTorrent bundles are originated by Jumbo, and the magnet URI's are provided within the Catalog. Players' computers will attempt to retrieve asset bundles via torrents first; if, however, a torrent source is not available rapidly enough, they will fall back upon fetching the assets from Jumbo directly over HTTP. This, in turn, makes them become an available source from which other players can later retrieve the same asset bundle.