The Teraflow Testbed

The Teraflow Testbed(TFT) is an infrastructure designed to use new 10 GB/s network protocols and data services for long haul, high performance networks. It consists of distributed nodes over three continents that can transmit, process, and mine very high volume data flows, or what we call teraflows. The TFT targets 10Gb/s wide area networks. The nodes are integrated using advanced 10 Gb/s photonic networks and rely on both Layer 2 optical switching and Layer 3 routers. Currently nodes are located in CERN, Switzerland; Amsterdam, The Netherlands; Tokyo, Japan; Chicago-National Center for Data Mining (NCDM), USA and Chicago-StarLight, USA. It is anticipated that nodes will be added in additional locations in the near future.

Using the TFT, NCDM is developing new network protocols and innovative web-based data integration and data mining services that scale to teraflows. We are also designing a new class of applications that move not only the queries and computations, but also the data when required. In addition, the TFT will be used to utilize these protocols and services over both traditional routed networks as well as lambda grids.

The TFT is designed to support high end-to-end transfers of data from a disk at one of the nodes to a disk at any of the other nodes. The TFT is also designed to perform data integration of two high volume data streams. Finally, the TFT is designed to allow parallelism of network flows within a single node, as well as striping across multiple nodes.