IceShare

Help Wanted

Hello Slashdotters - just to preceed all this juicy info on what IceShare is, I'd like to make a request to help get it finished sooner. We (Xiph foundation) could really use some help from people with one or more of the following skills:

• technical documents writter (for libraries, protocols, etc)
• player integration - getting this stuff available to users
• crypto guru - the IceShare system needs some help here
• python programmers - to help complete the proof of concept work

Please contact arc "at" Xiph "d0t" org if you can help. Thanks!

What is it?

IceShare is library that distributes Ogg streams on a pseudo-P2P network. It is heavily based on BitTorrent, but works on the Ogg page level, and unlike PeerCast it works with files as well as continuous streams.

It's designed to allow musicians, video producers, radio and television stations, or anyone looking to inexpensivly distribute audio/video on the web. It's intended to be initiated from websites, with links to icet:// URLs. It is not designed for P2P searching, such as Gnutella, Kazaa, and eDonkey provide, however websites may be setup to easily search content on one or more IceTracker servers.

Overview

IceShare is called pseudo-P2P because the network relies on a traditional client-server model for managing transfers between IceShare peers on the network.

The media players are the level which P2P takes place, whereas listeners who have available upstream bandwidth can help distribute the same content they're listening to amoung other listeners. This helps Icecast servers non-linearly scale for much larger listener loads and reduces the bandwidth requirements for hosting static Ogg multimedia on websites.

The IceShare library allows these features to be easily added to media players, including support for seeking to "not downloaded yet" parts of the media and available bandwidth detection/reporting for multi-bitrate streams.

IceTracker is a server that keeps track of who's activly sharing certain media and each of their send/receive ratios. IceTracker helps direct IceShare users to better hosts and track individual user's bandwidth and level of participation to reward high bandwidth/participation users with faster peers. IceTracker servers track users anonymously by a DSA key generated by each IceShare client.

Icecast connects to an IceTracker as a client to provide live stream information (pageno's, checksums, etc) and to receive guidance as per dropping less participating listeners when bandwidth is tight.

Media Players

URLs in the form icet://<icetracker>:<port>/<media> direct the media player to connect to an IceTracker using IceT protocol via the IceShare library. IceShare will state that it need's the specified media.

The IceTracker for that media should then respond with general information about the media in question, how many pages it has, how long its playtime is (or if it is continuous), and generally how long it should take to transfer it. This information should allow the media player to setup the seek bar and know how much it should buffer before beginning play.

IceTracker should then start directing IceShare to hosts which pieces of the media can be accessed from. IceShare does not know how much of the media each of those hosts has, since many may have only partial transfers. IceTracker specifies which page, or set of pages, to download from each host. IceShare responds with a quick "I got it" for each page, thereby letting IceTracker know that the reported page is ready to be shared with others. This also helps IceTracker keep track of latency and bandwidth between peers so that it can provide the client with better hosts.

If the player seeks to an not-yet-downloaded part of the media IceShare can express this to IceTracker, which will change its transfer focus to the seek point and beyond. In this way, especially for long pieces of media, the whole file does not have to be transfered to access a specific section of it.

IceShare also provides media players access to its "page table". The media player can use this to reflect media transfer stats in the seek bar, prehaps using an alternative background color to indicate sections of the media which have been downloaded.

IceShare handles incomming <A HREF="IceHTTP">HTTP</A> connections from peers, information about uploads on the P2P network are available to the media player but are not nessesary. The media player can tune the level of participation, limiting the amount of bandwidth or length of time a piece of media is available. For the most part, it's in the user's interest to participate as much as they're able to, since this will earn them faster access to other media through the same IceTracker.

A slightly-extended <A HREF="IceHTTP">HTTP/1.1</A> is used to specify page-ranges. IceShare should also support byte-ranges for traditional HTTP download agents which are attempting to resume a lost transfer.

Media Distributors

IceShare can also be used to distribute original media on the P2P network. A distribution client can use IceShare to connect to an IceTracker and inform it of the new media's statistics. This client should have enough upstream bandwidth to send the first few copies by itself, after which those who have downloaded it should begin sharing the load.

Icecast is a good example of a distribution client. It can use IceShare to inform IceTracker of its streams and continue to send it page information for each of its ongoing streams. Icecast servers using IceShare will still need enough bandwidth to send atleast one (preferably more) streams to listeners who can then redistribute it to other listeners.

IceTracker will allow IceShare clients to request current listeners and total "hits" for any media that it is tracking. This can be used by Icecast to accurately track listeners.

Alternative Streams

IceShare also includes support for alternative bitrates and codecs to published media. These alternatives can be used to meet the needs of each individual user on the network. For instance, a stream could be provided in 64kbps Vorbis, 24kbps Speex, and 24kbps Vorbis (in that order). Those with enough bandwidth will receive the default 64kbps Vorbis stream, while modem users will switch to either the Speex or the low bitrate Vorbis based on their ability to support Speex. This makes it possible for every IceShare peer to receive a continuous stream in the highest quality format their software and network connection allows them.

New Website

The new project website for IceShare is http://www.iceshare.org/ ! J_Bullet has volunteered to design the site, but it's just general information for now.

What's the Holdup?

While we're apparently fairly close to wrapping up this baby and start into some massive plugin coding, there's vital things missing from other Xiph projects which this needs. Specifically, OggFile granule handling needs to be decided on so we can get the IceT timing specs to match, we need to get the final scoop on discontinuous bitstreams, and a fully functional public tracker needs to be written (even if it's just in Python for now). Plus, it makes alot of sense for the plugins to be packaged with OggFile as a join distribution effort. Current estimates, given the committments from various Xiph developers and pace of development, is IceShare will begin getting deployed summer 2004. Plus, as revolutionary as it is, it'll take a few months for it's useage to be widespread enough to useful for general usage until atleast the end of 2004.

If you're reading this page and thinking "damned, that's awesome!" and want to speed up the above timeline we can always use volunteers. Even if you don't know any code, but prehaps you're good at HTML or graphic design, or can do no more than help test the system, we could use your help. Email Arc Riley <arc@xiph.org> for more information.

Discussion

I've moved discussion to the Talk page to keep load times down. Please add comments to Talk:IceShare. Thanks :-) -- Arc