Improved index compression techniques for versioned document collections

Jinru He, Junyuan Zeng, Torsten Suel

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Current Information Retrieval systems use inverted index structures for efficient query processing. Due to the extremely large size of many data sets, these index structures are usually kept in compressed form, and many techniques for optimizing compressed size and query processing speed have been proposed. In this paper, we focus on versioned document collections, that is, collections where each document is modified over time, resulting in multiple versions of the document. Consecutive versions of the same document are often similar, and several researchers have explored ideas for exploiting this similarity to decrease index size. We propose new index compression techniques for versioned document collections that achieve reductions in index size over previous methods. In particular, we first propose several bitwise compression techniques that achieve a compact index structure but that are too slow for most applications. Based on the lessons learned, we then propose additional techniques that come close to the sizes of the bitwise technique while also improving on the speed of the best previous methods.

    Original languageEnglish (US)
    Title of host publicationCIKM'10 - Proceedings of the 19th International Conference on Information and Knowledge Management and Co-located Workshops
    Pages1239-1248
    Number of pages10
    DOIs
    StatePublished - 2010
    Event19th International Conference on Information and Knowledge Management and Co-located Workshops, CIKM'10 - Toronto, ON, Canada
    Duration: Oct 26 2010Oct 30 2010

    Publication series

    NameInternational Conference on Information and Knowledge Management, Proceedings

    Other

    Other19th International Conference on Information and Knowledge Management and Co-located Workshops, CIKM'10
    CountryCanada
    CityToronto, ON
    Period10/26/1010/30/10

    Keywords

    • Index compression
    • Inverted index
    • Versioned documents

    ASJC Scopus subject areas

    • Decision Sciences(all)
    • Business, Management and Accounting(all)

    Fingerprint Dive into the research topics of 'Improved index compression techniques for versioned document collections'. Together they form a unique fingerprint.

    Cite this