What's hot and what's not: Tracking most frequent items dynamically

Graham Cormode, S. Muthukrishnan

    Research output: Contribution to journalReview article

    Abstract

    Most database management systems maintain statistics on the underlying relation. One of the important statistics is that of the "hot items" in the relation: those that appear many times (most frequently, or more than some threshold). For example, end-biased histograms keep the hot items as part of the histogram and are used in selectivity estimation. Hot items are used as simple outliers in data mining, and in anomaly detection in many applications. We present new methods for dynamically determining the hot items at any time in a relation which is undergoing deletion operations as well as inserts. Our methods maintain small space data structures that monitor the transactions on the relation, and, when required, quickly output all hot items without rescanning the relation in the database. With user-specified probability, all hot items are correctly reported. Our methods rely on ideas from "group testing." They are simple to implement, and have provable quality, space, and time guarantees. Previously known algorithms for this problem that make similar quality and performance guarantees cannot handle deletions, and those that handle deletions cannot make similar guarantees without rescanning the database. Our experiments with real and synthetic data show that our algorithms are accurate in dynamically tracking the hot items independent of the rate of insertions and deletions.

    Original languageEnglish (US)
    Pages (from-to)249-278
    Number of pages30
    JournalACM Transactions on Database Systems
    Volume30
    Issue number1
    DOIs
    StatePublished - Mar 2005

    Keywords

    • Approximate query answering
    • Data stream processing

    ASJC Scopus subject areas

    • Information Systems

    Fingerprint Dive into the research topics of 'What's hot and what's not: Tracking most frequent items dynamically'. Together they form a unique fingerprint.

    Cite this