Optimized inverted list assignment in distributed search engine architectures

Jiangong Zhang, Torsten Suei

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    We study efficient query processing in distributed web search engines with global index organization. The main performance bottleneck in this case is due to the large amount of index data that is exchanged between nodes during the processing of a query, and previous work has proposed several techniques for significantly reducing this cost. We describe an approach that provides substantial additional improvement over previous techniques. In particular, we analyze search engine query traces in order to optimize the assignment of index data to the nodes in the system, such that terms frequently occurring together in queries are also often collocated on the same node. Our experiments show that in return for a modest factor increase in storage space, we can achieve a reduction in communication cost of an order of magnitude over the previous best techniques.

    Original languageEnglish (US)
    Title of host publicationProceedings - 21st International Parallel and Distributed Processing Symposium, IPDPS 2007; Abstracts and CD-ROM
    DOIs
    StatePublished - 2007
    Event21st International Parallel and Distributed Processing Symposium, IPDPS 2007 - Long Beach, CA, United States
    Duration: Mar 26 2007Mar 30 2007

    Publication series

    NameProceedings - 21st International Parallel and Distributed Processing Symposium, IPDPS 2007; Abstracts and CD-ROM

    Other

    Other21st International Parallel and Distributed Processing Symposium, IPDPS 2007
    CountryUnited States
    CityLong Beach, CA
    Period3/26/073/30/07

    ASJC Scopus subject areas

    • Hardware and Architecture
    • Software
    • Mathematics(all)

    Fingerprint Dive into the research topics of 'Optimized inverted list assignment in distributed search engine architectures'. Together they form a unique fingerprint.

  • Cite this

    Zhang, J., & Suei, T. (2007). Optimized inverted list assignment in distributed search engine architectures. In Proceedings - 21st International Parallel and Distributed Processing Symposium, IPDPS 2007; Abstracts and CD-ROM [4227959] (Proceedings - 21st International Parallel and Distributed Processing Symposium, IPDPS 2007; Abstracts and CD-ROM). https://doi.org/10.1109/IPDPS.2007.370231