The Wambo File System
Wambo Blog | Thursday, November 23rd, 2006 | Trackback
Wambo is best known for its product, Swapper, which lets users compress and share their media publicly (via a profile page) or privately. Swapper (and early messaging prototypes Wambo has developed in the past) leverages the Wambo File System (WFS), a unique technology that enables large file transfers with virtually no infrastructure cost. Without the WFS, it would be impossible to deliver Swapper’s great benefits without running and maintaining hundreds, if not thousands of servers. So what’s the WFS and how does it work?
The WFS is the first hybrid P2P network to cleverly combine the latest compression, error correction and peer-swarming technologies to massively reduce bandwidth and infrastructure costs. Whether deployed as a peer-assisted CDN, or as the pending-pending infrastructure-less peer CDN, the Wambo File System provides unbeatable performance and cost efficiency in all content delivery scenarios.
In the first deployment model, called peer-assisted CDN, bandwidth and storage costs are balanced between centrally hosted servers and client peers. When transferring files via the Wambo File System (“WFS”), such as video files, the content is initially pre-processed, uploaded and stored to a hosted server. When the receiving-end triggers a download event, the fragments needed to rebuild the file are retrieved from the hosted storage servers, and other peer clients that have a local copy of the same file. Here are the steps in greater detail:
1. Pre-processing. The upload preparation is a critical phase where content is transformed in order to minimize bandwidth consumption and efficiently organize how the file fragments will be transmitted and stored.
Files are initially compressed and resized in the case of media content (optional), which can reduce their size by up to 95%. Immediately thereafter, the compressed content is fragmented into smaller pieces, and error corrected for redundancy purposes (using the low-density parity check code implementation of Forward Error Correction). A tracker for each of the files is also generated.
2. Upload. Prior to initiating the file transfer process, the client software will automatically verify that the files have not been uploaded already using a secure and anonymous signature check. If one of the files already has been uploaded by any user of software, then it does not need to be re-uploaded again (results in bandwidth savings and performance improvements).
3. Storage. Content is stored on hosted servers controlled by Wambo. These storage servers are commodity machines running the WFS, and acting individual storage units. Each of the fragments is saved to a pre-determined server, according to the instructions contained in the file tracker. Further, since each of the files has been error corrected, it doesn’t matter if one of the servers goes offline: files can be reconstructed from a fraction of the original fragments.
4. Download. When the receiving client requests a download for the file, or if pre-fetching has started, the file tracker is used to locate the fragments needed on the storage servers, or additional peer computers which may have downloaded a copy of the same file already.
5. Swarming. A fraction of the content will be downloaded from peer computers that are online and have a copy of the file that’s been downloaded. Swarming enables additional bandwidth savings since it offloads the central servers. It also allows large popular files to be delivered at a near-zero cost and excellent performance.
6. Reconstruction. As the file is downloaded from its fragments, and since all files are error corrected, regular checks are made until enough fragments have been retrieved to rebuild the original file. When the download is considered complete, the content is decompressed, de-encrypted and presented to the receiving end.
In the second deployment model, called Peer CDN, the entire download and storage burden is taken by peer clients, leaving the central servers virtually idle (with the exception of file tracker transactions). In this type of deployment, which does not require central hosted storage, fragments are uploaded directly to a set of available peer computers.
To download the file, the receiver will use the file tracker to locate where each of the fragments are located, and initiate concurrent download operations until enough fragments have been retrieved to rebuild the file.
Since the WFS uses error correction (FEC), the file can be rapidly reconstructed even if some of the fragments are missing.
Media delivery costs. Compared to centralized delivery techniques, the Wambo’s peer-assisted delivery method lowers the bandwidth and storage costs for popular files. The peer CDN outperforms any other delivery techniques from the very first delivery, keeping the marginal cost per download constant for any audience scale.
