IceShare: Difference between revisions

Revision as of 02:38, 18 July 2006

The following is a draft!

It is at best incomplete and at worst completely broken. In any case, it is not an “official” Xiph spec or codec, so use with care.

Help Wanted

Just to preceed all this info on what IceShare is, I'd like to make a request to help get it finished sooner. We (Xiph foundation) could really use some help from people with one or more of the following skills:

Technical documents writer (for libraries, protocols, etc)
Crypto guru - the IceShare system needs some help with hashing and encryption in general
Python programmers - to help complete the prototype suite
Player integration - getting this system available to users

Please contact Arc <arc@Xiph.org> if you can help. Thanks!

What is it?

IceShare is library that distributes Ogg streams on a pseudo-P2P network. It is heavily based on BitTorrent, but works on the Ogg page level, and unlike PeerCast it works with files as well as continuous streams.

It's designed to allow musicians, video producers, radio and television stations, or anyone looking to inexpensivly distribute audio/video on the web. It's intended to be initiated from websites, with links to icet:// URLs. It is not designed for P2P searching, such as Gnutella, Kazaa, and eDonkey provide, however websites may be setup to easily search content on one or more IceTracker servers.

Overview

IceShare is called pseudo-P2P because the network relies on a traditional client-server model for managing transfers between IceShare peers on the network.

The media players are the level which P2P takes place, whereas listeners who have available upstream bandwidth can help distribute the same content they're listening to amoung other listeners. This helps Icecast servers non-linearly scale for much larger listener loads and reduces the bandwidth requirements for hosting static Ogg multimedia on websites.

The IceShare library allows these features to be easily added to media players, including support for seeking to "not downloaded yet" parts of the media and available bandwidth detection/reporting for multi-bitrate streams.

IceTracker is a server that keeps track of who's actively sharing certain media and each of their send/receive ratios. IceTracker helps direct IceShare users to better hosts and track individual user's bandwidth and level of participation to reward high bandwidth/participation users with faster peers. IceTracker servers track users anonymously by a DSA key generated by each IceShare client.

Icecast connects to an IceTracker as a client to provide live stream information (pageno's, checksums, etc) and to receive guidance as per dropping less participating listeners when bandwidth is tight.

Media Players

URLs in the form icet://<icetracker>:<port>/<media> direct the media player to connect to an IceTracker using IceT protocol via the IceShare library. IceShare will state that it need's the specified media.

The IceTracker for that media should then respond with general information about the media in question, how many pages it has, how long its playtime is (or if it is continuous), and generally how long it should take to transfer it. This information should allow the media player to setup the seek bar and know how much it should buffer before beginning play.

IceTracker should then start directing IceShare to hosts which pieces of the media can be accessed from. IceShare does not know how much of the media each of those hosts has, since many may have only partial transfers. IceTracker specifies which page, or set of pages, to download from each host. IceShare responds with a quick "I got it" for each page, thereby letting IceTracker know that the reported page is ready to be shared with others. This also helps IceTracker keep track of latency and bandwidth between peers so that it can provide the client with better hosts.

If the player seeks to an not-yet-downloaded part of the media IceShare can express this to IceTracker, which will change its transfer focus to the seek point and beyond. In this way, especially for long pieces of media, the whole file does not have to be transfered to access a specific section of it.

IceShare also provides media players access to its "page table". The media player can use this to reflect media transfer stats in the seek bar, prehaps using an alternative background color to indicate sections of the media which have been downloaded.

IceShare handles incoming <A HREF="IceHTTP">HTTP</A> connections from peers, information about uploads on the P2P network are available to the media player but are not nessesary. The media player can tune the level of participation, limiting the amount of bandwidth or length of time a piece of media is available. For the most part, it's in the user's interest to participate as much as they're able to, since this will earn them faster access to other media through the same IceTracker.

A slightly-extended <A HREF="IceHTTP">HTTP/1.1</A> is used to specify page-ranges. IceShare should also support byte-ranges for traditional HTTP download agents which are attempting to resume a lost transfer.

Media Distributors

IceShare can also be used to distribute original media on the P2P network. A distribution client can use IceShare to connect to an IceTracker and inform it of the new media's statistics. This client should have enough upstream bandwidth to send the first few copies by itself, after which those who have downloaded it should begin sharing the load.

Icecast is a good example of a distribution client. It can use IceShare to inform IceTracker of its streams and continue to send it page information for each of its ongoing streams. Icecast servers using IceShare will still need enough bandwidth to send atleast one (preferably more) streams to listeners who can then redistribute it to other listeners.

IceTracker will allow IceShare clients to request current listeners and total "hits" for any media that it is tracking. This can be used by Icecast to accurately track listeners.

Alternative Streams

IceShare also includes support for alternative bitrates and codecs to published media. These alternatives can be used to meet the needs of each individual user on the network. For instance, a stream could be provided in 64kbps Vorbis, 24kbps Speex, and 24kbps Vorbis (in that order). Those with enough bandwidth will receive the default 64kbps Vorbis stream, while modem users will switch to either the Speex or the low bitrate Vorbis based on their ability to support Speex. This makes it possible for every IceShare peer to receive a continuous stream in the highest quality format their software and network connection allows them.

Payload Protocols

One of the interesting things about IceShare is that it's designed to use many different protocols for the actual file transfer. This allows a combination of protocols to be used, even in between the same two hosts, in the effort to get the media deployed in the most efficient manner possible.

This also allows IceShare to be combined with other P2P systems. For instance, if someone chose to ignore PeerCast's GPL license addendum which doesn't allow modified clients to connect to their metaserver, a broadcaster could choose to stream to both Peercast and IceShare whereas any IceShare peer with a Peercast plugin could be sent to grab parts or all of the stream from that P2P network. The same is also true for BitTorrent, Gnutella, or any other P2P system.

Here are the requirements for an IceShare Payload Protocol:

You must be able to request a path (local URL) for a media
- Each media must be at the same path for every protocol
- Local media path sorting and arranging is up to the local implementation
  - All media may be in one directory or many, even layers deep
  - Media may be moved locally (after notifying the Tracker)
You must be able to specify a range of data for download
- Protocols supporting only Byte ranges limits it's useage
  - A peer may not nessesarily know where in a given binary stream a series of pages belong
  - Byte ranges are only known for pages in a continuous series from the start of the media
- Time ranges are more useable, but may result in wasted bandwidth
- Supporting (Ogg) Page ranges allows total useability
- Range can be requested in any way nessesary for the protocol
  - Some payload protocols may even append it to the media's path
Binary data must be able to be transfered by some means
- Ogg Pages do not need to be seperated in this data
  - libogg2 provides a fast and efficient manner for seperating Pages by the receiver
- Delivery of any segment does not need to be guarenteed
  - The IceTracker will make sure everyone gets a specific Page
  - Retransmittial by IceTracker is only supported with Page granularity
- Error detection does not need to be implemented
  - Each Ogg Page has it's own CRC checksum in it's header
- Order should be guarenteed for data within a single Ogg Page
  - Ogg Pages may be sent complete in a single packet for this
  - Many packets may carry sequential Pages with packet ordering provided in a small header
The IceTracker does not need to know any details of a payload protocol
- It will learn new protocol names when support is advertised by peers
- It will learn their strengths and weaknesses by building statistical data
- Protocols will be used based on the bandwidth, reliability, and timing needs of a peer

Security Model

Three layers of "security" are provided by the IceShare system to ensure data is transfered without errors, without alteration by peers, and without alteration of trusted content from trackers.

What's the Holdup?

While we're apparently fairly close to wrapping up this baby and start into some massive plugin coding, there's vital things missing from other Xiph projects which this needs.

Specifically:

~~OggFile~~ OggStream granule handling needs to be decided on so we can get the IceT timing specs to match
~~we need to get the final scoop on discontinuous bitstreams~~ done and integrated into libogg2
a fully functional public tracker needs to be written (even if it's just in Python for now).
it makes alot of sense for the plugins to be packaged with ~~OggFile~~ OggStream as a join distribution effort.

Current estimates, given the committments from various Xiph developers and pace of development, is IceShare will begin getting deployed summer ~~2004~~ 2005.

Updated: 1/10/05 -- I've had to delay IceShare in order to get OggStream (formerly OggFile) ready. This explains why this project is so over-due.

If you're reading this page and thinking "damned, that's awesome!" and want to speed up the above timeline we can always use volunteers. Even if you don't know any code, but prehaps you're good at HTML or graphic design, or can do no more than help test the system, we could use your help. Email Arc <arc@xiph.org> for more information.

Discussion

Discussion has been moved to the Talk page to keep load times down. Please add comments to Talk:IceShare.