Query-aware sampling for data streams

Theodore Johnson, S. Muthukrishnan, Vladislav Shkapenyuk, Oliver Spatscheck

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Data Stream Management Systems are useful when large volumes of data need to be processed in real time. Examples include monitoring network traffic, monitoring financial transactions, and analyzing large scale scientific data feeds. These applications have varying data rates and often show bursts of high activity that overload the system, often during the most critical instants (e.g., network attacks, financial spikes) for analysis. Therefore, load shedding is necessary to preserve the stability of the system, gracefully degrade its performance and extract answers. Existing methods for load shedding in a general purpose data stream query system use random sampling of tuples, essentially independent of the query. While this technique is acceptable for some queries, the results may be meaningless or even incorrect for other queries. In principle, a number of different query-dependent sampling methods exist, but they work only for particular queries. In this paper, we show how to perform query-aware sampling (semantic sampling) which works in general. We present methods for analyzing any given query, choosing sampling methods judiciously, and reconciling the sampling methods required by different queries in a query set. We conclude with experiments on a high-speed data stream that demonstrate with different query sets that our method produces accurate results while decreasing the load significantly.

    Original languageEnglish (US)
    Title of host publicationWorkshops in Conjunction with the International Conference on Data Engineering - ICDE' 07
    Pages664-673
    Number of pages10
    DOIs
    StatePublished - 2007
    EventWorkshops in Conjunction with the 23rd International Conference on Data Engineering - ICDE 2007 - Istanbul, Turkey
    Duration: Apr 15 2007Apr 20 2007

    Publication series

    NameProceedings - International Conference on Data Engineering
    ISSN (Print)1084-4627

    Conference

    ConferenceWorkshops in Conjunction with the 23rd International Conference on Data Engineering - ICDE 2007
    Country/TerritoryTurkey
    CityIstanbul
    Period4/15/074/20/07

    ASJC Scopus subject areas

    • Software
    • Signal Processing
    • Information Systems

    Fingerprint

    Dive into the research topics of 'Query-aware sampling for data streams'. Together they form a unique fingerprint.

    Cite this