A heartbeat mechanism and its application in gigascope

Theodore Johnson, S. Muthukrishnan, Vladislav Shkapenyuk, Oliver Spatscheck

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Data stream management systems often rely on ordering properties of tuple attributes in order to implement non-blocking operators. However, query operators that work with multiple streams, such as stream merge or join, can often still block if one of the input stream is very slow or bursty. In principle, punctuation and heartbeat mechanisms have been proposed to unblock streaming operators. In practice, it is a challenge to incorporate such mechanisms into a high-performance stream management system that is operational in an industrial application. In this paper, we introduce a system for punctuation-carrying heartbeat generation that we developed for Gigascope, a high-performance streaming database for network monitoring, that is operationally used within AT&T's IP backbone. We show how heartbeats can be regularly generated by low-level nodes in query execution plans and propagated upward unblocking all streaming operators on its way. Additionally, our heartbeat mechanism can be used for other applications in distributed settings such as detecting node failures, performance monitoring, and query optimization. A performance evaluation using live data feeds shows that our system is capable of working at multiple Gigabit line speeds in a live, industrial deployment and can significantly decrease the query memory utilization.

    Original languageEnglish (US)
    Title of host publicationVLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases
    Pages1079-1088
    Number of pages10
    StatePublished - 2005
    EventVLDB 2005 - 31st International Conference on Very Large Data Bases - Trondheim, Norway
    Duration: Aug 30 2005Sep 2 2005

    Publication series

    NameVLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases
    Volume3

    Other

    OtherVLDB 2005 - 31st International Conference on Very Large Data Bases
    CountryNorway
    CityTrondheim
    Period8/30/059/2/05

    ASJC Scopus subject areas

    • Engineering(all)

    Fingerprint Dive into the research topics of 'A heartbeat mechanism and its application in gigascope'. Together they form a unique fingerprint.

    Cite this