TY - GEN
T1 - Accelerating data for compute clusters and grids
AU - Marchand, Benoit
PY - 2006
Y1 - 2006
N2 - Effective scaling and maximal throughput for clusters is an ongoing issue for computational workloads. We will discuss an approach to workload distribution which is data-centric, rather than process-centric. Data is moved to nodes ahead of computation. The workload management is handled as usual, except that the presence of data at nodes is used to trigger process scheduling. We will discuss how to create workflows which are data-activated and how an asynchronous pipeline can be established, allowing file serving latency to be hidden.In addition our RepliCator solution provides broadcasting to all nodes in a cluster simultaneously, substantially reducing data transfer times. This can be highly effective for throughput problems. We will present benchmark results obtained on clusters with several hundred cpus. Speedups in overall processing of factors of 2.5 to 4.5 have been observed using the combination of data-activated processing and data broadcasting provided by RepliCator data transfer management software.
AB - Effective scaling and maximal throughput for clusters is an ongoing issue for computational workloads. We will discuss an approach to workload distribution which is data-centric, rather than process-centric. Data is moved to nodes ahead of computation. The workload management is handled as usual, except that the presence of data at nodes is used to trigger process scheduling. We will discuss how to create workflows which are data-activated and how an asynchronous pipeline can be established, allowing file serving latency to be hidden.In addition our RepliCator solution provides broadcasting to all nodes in a cluster simultaneously, substantially reducing data transfer times. This can be highly effective for throughput problems. We will present benchmark results obtained on clusters with several hundred cpus. Speedups in overall processing of factors of 2.5 to 4.5 have been observed using the combination of data-activated processing and data broadcasting provided by RepliCator data transfer management software.
UR - http://www.scopus.com/inward/record.url?scp=34548209952&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=34548209952&partnerID=8YFLogxK
U2 - 10.1145/1188455.1188731
DO - 10.1145/1188455.1188731
M3 - Conference contribution
AN - SCOPUS:34548209952
SN - 0769527000
SN - 9780769527000
T3 - Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06
BT - Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06
ER -