Query-aware sampling for data streams

Data Stream Management Systems are useful when large volumes of data need to be processed in real time. Examples include monitoring network traffic, monitoring financial transactions, and analyzing large scale scientific data feeds. These applications have varying data rates and often show bursts of high activity that overload the system, often during the most critical instants (e.g., network attacks, financial spikes) for analysis. Therefore, load shedding is necessary to preserve the stability of the system, gracefully degrade its performance and extract answers. Existing methods for load shedding in a general purpose data stream query system use random sampling of tuples, essentially independent of the query. While this technique is acceptable for some queries, the results may be meaningless or even incorrect for other queries. In principle, a number of different query-dependent sampling methods exist, but they work only for particular queries. In this paper, we show how to perform query-aware sampling (semantic sampling) which works in general. We present methods for analyzing any given query, choosing sampling methods judiciously, and reconciling the sampling methods required by different queries in a query set. We conclude with experiments on a high-speed data stream that demonstrate with different query sets that our method produces accurate results while decreasing the load significantly.