What's different: Distributed, continuous monitoring of duplicate-resilient aggregates on data streams

Graham Cormode, S. Muthukrishnan, Wei Zhuang

    Research output: Chapter in Book/Report/Conference proceedingConference contribution


    Emerging applications in sensor systems and network-wide IP traffic analysis present many technical challenges. They need distributed monitoring and continuous tracking of events. They have severe resource constraints not only at each site in terms of per-update processing time and archival space for high-speed streams of observations, but also crucially, communication constraints for collaborating on the monitoring task. These elements have been addressed in a series of recent works. A fundamental issue that arises is that one cannot make the "uniqueness" assumption on observed events which is present in previous works, since widescale monitoring invariably encounters the same events at different points. For example, within the network of an Internet Service Provider packets of the same flow will be observed in different routers; similarly, the same individual will be observed by multiple mobile sensors in monitoring wild animals. Aggregates of interest on such distributed environments must be resilient to duplicate observations. We study such duplicate-resilient aggregates that measure the extent of the duplication - how many unique observations are there, how many observations are unique - as well as standard holistic aggregates such as quantiles and heavy hitters over the unique items. We present accuracy guaranteed, highly communication-efficient algorithms for these aggregates that work within the time and space constraints of high speed streams. We also present results of a detailed experimental study on both real-life and synthetic data.

    Original languageEnglish (US)
    Title of host publicationProceedings of the 22nd International Conference on Data Engineering, ICDE '06
    Number of pages1
    StatePublished - 2006
    Event22nd International Conference on Data Engineering, ICDE '06 - Atlanta, GA, United States
    Duration: Apr 3 2006Apr 7 2006

    Publication series

    NameProceedings - International Conference on Data Engineering
    ISSN (Print)1084-4627


    Other22nd International Conference on Data Engineering, ICDE '06
    Country/TerritoryUnited States
    CityAtlanta, GA

    ASJC Scopus subject areas

    • Software
    • Signal Processing
    • Information Systems


    Dive into the research topics of 'What's different: Distributed, continuous monitoring of duplicate-resilient aggregates on data streams'. Together they form a unique fingerprint.

    Cite this