The power of choice in data-aware cluster scheduling

Shivaram Venkataraman; Aurojit Panda; Ganesh Ananthanarayanan; Michael J. Franklin; Ion Stoica

The power of choice in data-aware cluster scheduling

Shivaram Venkataraman, Aurojit Panda, Ganesh Ananthanarayanan, Michael J. Franklin, Ion Stoica

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Providing timely results in the face of rapid growth in data volumes has become important for analytical frameworks. For this reason, frameworks increasingly operate on only a subset of the input data. A key property of such sampling is that combinatorially many subsets of the input are present. We present KMN, a system that leverages these choices to perform data-aware scheduling, i.e., minimize time taken by tasks to read their inputs, for a DAG of tasks. KMN not only uses choices to co-locate tasks with their data but also percolates such combinatorial choices to downstream tasks in the DAG by launching a few additional tasks at every upstream stage. Evaluations using workloads from Facebook and Conviva on a 100-machine EC2 cluster show that KMN reduces average job duration by 81% using just 5% additional resources.

Original language	English (US)
Title of host publication	Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014
Publisher	USENIX Association
Pages	301-316
Number of pages	16
ISBN (Electronic)	9781931971164
State	Published - Jan 1 2014
Event	11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014 - Broomfield, United States Duration: Oct 6 2014 → Oct 8 2014

Publication series

Name	Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014

Conference

Conference	11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014
Country	United States
City	Broomfield
Period	10/6/14 → 10/8/14

ASJC Scopus subject areas

Information Systems
Computer Networks and Communications
Hardware and Architecture

Fingerprint Dive into the research topics of 'The power of choice in data-aware cluster scheduling'. Together they form a unique fingerprint.

Cite this

Venkataraman, S., Panda, A., Ananthanarayanan, G., Franklin, M. J., & Stoica, I. (2014). The power of choice in data-aware cluster scheduling. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014 (pp. 301-316). (Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014). USENIX Association.

The power of choice in data-aware cluster scheduling. / Venkataraman, Shivaram; Panda, Aurojit; Ananthanarayanan, Ganesh; Franklin, Michael J.; Stoica, Ion.

Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014. USENIX Association, 2014. p. 301-316 (Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Venkataraman, S, Panda, A, Ananthanarayanan, G, Franklin, MJ & Stoica, I 2014, The power of choice in data-aware cluster scheduling. in Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014. Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014, USENIX Association, pp. 301-316, 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014, Broomfield, United States, 10/6/14.

Venkataraman, Shivaram ; Panda, Aurojit ; Ananthanarayanan, Ganesh ; Franklin, Michael J. ; Stoica, Ion. / The power of choice in data-aware cluster scheduling. Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014. USENIX Association, 2014. pp. 301-316 (Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014).

@inproceedings{a8713172c7f44bb58a5baa79442f3a9d,

title = "The power of choice in data-aware cluster scheduling",

abstract = "Providing timely results in the face of rapid growth in data volumes has become important for analytical frameworks. For this reason, frameworks increasingly operate on only a subset of the input data. A key property of such sampling is that combinatorially many subsets of the input are present. We present KMN, a system that leverages these choices to perform data-aware scheduling, i.e., minimize time taken by tasks to read their inputs, for a DAG of tasks. KMN not only uses choices to co-locate tasks with their data but also percolates such combinatorial choices to downstream tasks in the DAG by launching a few additional tasks at every upstream stage. Evaluations using workloads from Facebook and Conviva on a 100-machine EC2 cluster show that KMN reduces average job duration by 81% using just 5% additional resources.",

author = "Shivaram Venkataraman and Aurojit Panda and Ganesh Ananthanarayanan and Franklin, {Michael J.} and Ion Stoica",

year = "2014",

month = jan,

day = "1",

language = "English (US)",

series = "Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014",

publisher = "USENIX Association",

pages = "301--316",

booktitle = "Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014",

note = "11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014 ; Conference date: 06-10-2014 Through 08-10-2014",

}

TY - GEN

T1 - The power of choice in data-aware cluster scheduling

AU - Venkataraman, Shivaram

AU - Panda, Aurojit

AU - Ananthanarayanan, Ganesh

AU - Franklin, Michael J.

AU - Stoica, Ion

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Providing timely results in the face of rapid growth in data volumes has become important for analytical frameworks. For this reason, frameworks increasingly operate on only a subset of the input data. A key property of such sampling is that combinatorially many subsets of the input are present. We present KMN, a system that leverages these choices to perform data-aware scheduling, i.e., minimize time taken by tasks to read their inputs, for a DAG of tasks. KMN not only uses choices to co-locate tasks with their data but also percolates such combinatorial choices to downstream tasks in the DAG by launching a few additional tasks at every upstream stage. Evaluations using workloads from Facebook and Conviva on a 100-machine EC2 cluster show that KMN reduces average job duration by 81% using just 5% additional resources.

AB - Providing timely results in the face of rapid growth in data volumes has become important for analytical frameworks. For this reason, frameworks increasingly operate on only a subset of the input data. A key property of such sampling is that combinatorially many subsets of the input are present. We present KMN, a system that leverages these choices to perform data-aware scheduling, i.e., minimize time taken by tasks to read their inputs, for a DAG of tasks. KMN not only uses choices to co-locate tasks with their data but also percolates such combinatorial choices to downstream tasks in the DAG by launching a few additional tasks at every upstream stage. Evaluations using workloads from Facebook and Conviva on a 100-machine EC2 cluster show that KMN reduces average job duration by 81% using just 5% additional resources.

UR - http://www.scopus.com/inward/record.url?scp=84971514453&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84971514453&partnerID=8YFLogxK

M3 - Conference contribution

T3 - Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014

SP - 301

EP - 316

BT - Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014

PB - USENIX Association

T2 - 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014

Y2 - 6 October 2014 through 8 October 2014

ER -