Network design considerations for exascale supercomputers

We consider the network design optimization for the Exascale class supercomputers by altering the widely analyzed and implemented torus networks. Our alteration scheme involves interlacing the torus networks with bypass links of lengths 6 hops, 9 hops, 12 hops, and mixed 6 and 12 hops. These bypass links are optimal resulting from exhaustive search of massive possibilities. Our case study is constructed by strategically coupling 288 racks of 6 × 6 × 36 nodes to a full system with 72 × 72 × 72 nodes. The peak performance of such a system is 0.56 Exa-flops when CPU-GPU complexes are adopted as a node module capable of 1.5 Tflops. Our design optimizes, simultaneously, the system performance, performancecost ratio, and power efficiency. The network diameter and the average node-to-node network distance, regarded as the performance metrics, got reduced, from the original 3D torus network, by 83.3% and 80.4%, respectively. Similarly, the performance-cost ratio and power efficiency are also increased 1.43 and 4.44 times, respectively.