Spartan: A distributed array framework with smart tiling

Chien Chin Huang, Qi Chen, Zhaoguo Wang, Russell Power, Jorge Ortiz, Jinyang Li, Zhen Xiao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Application programmers in domains like machine learning, scientific computing, and computational biology are accustomed to using powerful, high productivity array languages such as MatLab, R and NumPy. Distributed array frameworks aim to scale array programs across machines. However, maximizing the locality of access to distributed arrays is an unsolved problem; such locality is critical for high performance. This paper presents Spartan, a distributed array framework that automatically determines how to best partition (aka "tile") ndimensional arrays and to co-locate data with computation to maximize locality. Spartan combines a lazy-evaluation based, optimizing frontend with a distributed tiled array backend. Central to Spartan's design is a small number of carefully chosen parallel high-level operators, which form the expression graph captured by Spartan's frontend during runtime. These operators simplify the programming of distributed applications. More importantly, their well-defined semantics allow Spartan's runtime to calculate the costs of different tiling strategies and pick the best one for evaluating the entire expression graph. Using Spartan, we have implemented 12 applications from a variety of domains including machine learning and scientific computing. Our evaluations show that Spartan's automatic tiling mechanism leads to good and scala.

Original languageEnglish (US)
Title of host publicationProceedings of the 2015 USENIX Annual Technical Conference, USENIX ATC 2015
PublisherUSENIX Association
Pages1-15
Number of pages15
ISBN (Electronic)9781931971225
StatePublished - 2015
Event2015 USENIX Annual Technical Conference, USENIX ATC 2015 - Santa Clara, United States
Duration: Jul 8 2015Jul 10 2015

Publication series

NameProceedings of the 2015 USENIX Annual Technical Conference, USENIX ATC 2015

Conference

Conference2015 USENIX Annual Technical Conference, USENIX ATC 2015
CountryUnited States
CitySanta Clara
Period7/8/157/10/15

ASJC Scopus subject areas

  • Computer Science(all)

Fingerprint Dive into the research topics of 'Spartan: A distributed array framework with smart tiling'. Together they form a unique fingerprint.

  • Cite this

    Huang, C. C., Chen, Q., Wang, Z., Power, R., Ortiz, J., Li, J., & Xiao, Z. (2015). Spartan: A distributed array framework with smart tiling. In Proceedings of the 2015 USENIX Annual Technical Conference, USENIX ATC 2015 (pp. 1-15). (Proceedings of the 2015 USENIX Annual Technical Conference, USENIX ATC 2015). USENIX Association.