Drizzle: Fast and Adaptable Stream Processing at Scale

Shivaram Venkataraman, Michael Armbrust, Aurojit Panda, Ali Ghodsi, Kay Ousterhout, Michael J. Franklin, Benjamin Recht, Ion Stoica

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Large scale streaming systems aim to provide high throughput and low latency. They are often used to run mission-critical applications, and must be available 24x7. Thus such systems need to adapt to failures and inherent changes in workloads, with minimal impact on latency and throughput. Unfortunately, existing solutions require operators to choose between achieving low latency during normal operation and incurring minimal impact during adaptation. Continuous operator streaming systems, such as Naiad and Flink, provide low latency during normal execution but incur high overheads during adaptation (e.g., recovery), while micro-batch systems, such as Spark Streaming and FlumeJava, adapt rapidly at the cost of high latency during normal operations. Our key observation is that while streaming workloads require millisecond-level processing, workload and cluster properties change less frequently. Based on this, we develop Drizzle, a system that decouples the processing interval from the coordination interval used for fault tolerance and adaptability. Our experiments on a 128 node EC2 cluster show that on the Yahoo Streaming Benchmark, Drizzle can achieve end-to-end record processing latencies of less than 100ms and can get 2–3x lower latency than Spark. Drizzle also exhibits better adaptability, and can recover from failures 4x faster than Flink while having up to 13x lower latency during recovery.

Original languageEnglish (US)
Title of host publicationSOSP 2017 - Proceedings of the 26th ACM Symposium on Operating Systems Principles
PublisherAssociation for Computing Machinery, Inc
Pages374-389
Number of pages16
ISBN (Electronic)9781450350853
DOIs
StatePublished - Oct 14 2017
Event26th ACM Symposium on Operating Systems Principles, SOSP 2017 - Shanghai, China
Duration: Oct 28 2017Oct 31 2017

Publication series

NameSOSP 2017 - Proceedings of the 26th ACM Symposium on Operating Systems Principles

Other

Other26th ACM Symposium on Operating Systems Principles, SOSP 2017
Country/TerritoryChina
CityShanghai
Period10/28/1710/31/17

Keywords

  • Performance
  • Reliability
  • Stream Processing

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Science Applications
  • Software

Fingerprint

Dive into the research topics of 'Drizzle: Fast and Adaptable Stream Processing at Scale'. Together they form a unique fingerprint.

Cite this