The case for tiny tasks in compute clusters

Kay Ousterhout, Aurojit Panda, Joshua Rosen, Shivaram Venkataraman, Reynold Xin, Sylvia Ratnasamy, Scott Shenker, Ion Stoica

Research output: Contribution to conferencePaperpeer-review

Abstract

We argue for breaking data-parallel jobs in compute clusters into tiny tasks that each complete in hundreds of milliseconds. Tiny tasks avoid the need for complex skew mitigation techniques: by breaking a large job into millions of tiny tasks, work will be evenly spread over available resources by the scheduler. Furthermore, tiny tasks alleviate long wait times seen in today's clusters for interactive jobs: even large batch jobs can be split into small tasks that finish quickly. We demonstrate a 5.2x improvement in response times due to the use of smaller tasks. In current data-parallel computing frameworks, high task launch overheads and scalability limitations prevent users from running short tasks. Recent research has addressed many of these bottlenecks; we discuss remaining challenges and propose a task execution framework that can efficiently support tiny tasks.

Original languageEnglish (US)
StatePublished - 2013
Event14th Workshop on Hot Topics in Operating Systems, HotOS 2013 - Santa Ana Pueblo, United States
Duration: May 13 2013May 15 2013

Conference

Conference14th Workshop on Hot Topics in Operating Systems, HotOS 2013
CountryUnited States
CitySanta Ana Pueblo
Period5/13/135/15/13

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture
  • Information Systems

Fingerprint Dive into the research topics of 'The case for tiny tasks in compute clusters'. Together they form a unique fingerprint.

Cite this