Modeling skew in data streams

Flip Korn, S. Muthukrishnan, Yihua Wu

    Research output: Chapter in Book/Report/Conference proceedingConference contribution


    Data stream applications have made use of statistical summaries to reason about the data using nonparametric tools such as histograms, heavy hitters, and join sizes. However, relatively little attention has been paid to modeling stream data parametrically, despite the potential this approach has for mining the data. The challenges to do model fitting at streaming speeds are both technical - how to continually find fast and reliable parameter estimates on high speed streams of skewed data using small space - and conceptual - how to validate the goodness-of-fit and stability of the model online.In this paper, we show how to fit hierarchical (binomial multifractal) and non-hierarchical (Pareto) power-law models on a data stream. We address the technical challenges using an approach that maintains a sketch of the data stream and fits least-squares straight lines; it yields algorithms that are fast, space-efficient, and provide approximations of parameter value estimates with a priori quality guarantees relative to those obtained offline. We address the conceptual challenge by designing fast methods for online goodness-of-fit measurements on a data stream; we adapt the statistical testing technique of examining the quantile-quantile (q-q) plot, to perform online model validation at streaming speeds.As a concrete application of our techniques, we focus on network traffic data which has been shown to exhibit skewed distributions. We complement our analytic and algorithmic results with experiments on IP traffic streams in AT&T's Gigascope data stream management system, to demonstrate practicality of our methods at line speeds. We measured the stability and robustness of these models over weeks of operational packet data in an IP network. In addition, we study an intrusion detection application, and demonstrate the potential of online parametric modeling.

    Original languageEnglish (US)
    Title of host publicationSIGMOD 2006 - Proceedings of the ACM SIGMOD International Conference on Management of Data
    Number of pages12
    StatePublished - 2006
    Event2006 ACM SIGMOD International Conference on Management of Data - Chicago, IL, United States
    Duration: Jun 27 2006Jun 29 2006

    Publication series

    NameProceedings of the ACM SIGMOD International Conference on Management of Data
    ISSN (Print)0730-8078


    Other2006 ACM SIGMOD International Conference on Management of Data
    Country/TerritoryUnited States
    CityChicago, IL


    • Estimation
    • Modeling
    • Skew
    • Streaming algorithms

    ASJC Scopus subject areas

    • Software
    • Information Systems


    Dive into the research topics of 'Modeling skew in data streams'. Together they form a unique fingerprint.

    Cite this