Mining deviants in time series data streams

S. Muthukrishnan, Rahul Shah, Jeffrey Scott Vitter

    Research output: Contribution to journalConference articlepeer-review

    Abstract

    One of the central tasks in managing, monitoring and mining data streams is that of identifying outliers. There is a long history of study of various outliers in statistics and databases, and a recent focus on mining outliers in data streams. Here, we adopt the notion of "deviants" from Jagadish et al [19] as outliers. Deviants are based on one of the most fundamental statistical concept of standard deviation (or variance). Formally, deviants are defined based on a representation sparsity metric, i.e., deviants are values whose removal from the dataset leads to an improved compressed representation of the remaining items. Thus, deviants are not global maxima/minima, but rather these are appropriate local aberrations. Deviants are known to be of great mining value in time series databases. We present first-known algorithms for identifying deviants on massive data streams. Our algorithms monitor streams using very small space (polylogarithmic in data size) and are able to quickly find deviants at any instant, as the data stream evolves over time. For all versions of this problem - uni- vs multivariate time series, optimal vs near-optimal vs heuristic solutions, offline vs streaming - our algorithms have the same framework of maintaining a hierarchical set of candidate deviants that are updated as the time series data gets progressively revealed. We show experimentally using real network traffic data (SNMP aggregate time series) as well as synthetic data that our algorithm is remarkably accurate in determining the deviants.

    Original languageEnglish (US)
    Pages (from-to)41-50
    Number of pages10
    JournalProceedings of the International Conference on Scientific and Statistical Database Management, SSDBM
    Volume16
    StatePublished - 2004
    EventProceedings - 16th International Conference on Scientific and Statistical Databse Management, SSDBM 2004 - Santorini Island, Greece
    Duration: Jun 21 2004Jun 23 2004

    ASJC Scopus subject areas

    • Software
    • Applied Mathematics

    Fingerprint Dive into the research topics of 'Mining deviants in time series data streams'. Together they form a unique fingerprint.

    Cite this