Batch query processing for web search engines

Shuai Ding, Ricardo Baeza-Yates, Josh Attenberg, Torsten Suel

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Large web search engines are now processing billions of queries per day. Most of these queries are interactive in nature, requiring a response in fractions of a second. However, there are also a number of important scenarios where large batches of queries are submitted for various web mining and system optimization tasks that do not require an immediate response. Given the significant cost of executing search queries over billions of web pages, it is a natural question to ask if such batches of queries can be more efficiently executed than interactive queries. In this paper, we motivate and discuss the problem of batch query processing in search engines, identify basic mechanisms for improving the performance of such queries, and provide a preliminary experimental evaluation of the proposed techniques. Our conclusion is that significant cost reductions are possible by using specialized mechanisms for executing batch queries in Web search engines.

    Original languageEnglish (US)
    Title of host publicationProceedings of the 4th ACM International Conference on Web Search and Data Mining, WSDM 2011
    Pages137-146
    Number of pages10
    DOIs
    StatePublished - 2011
    Event4th ACM International Conference on Web Search and Data Mining, WSDM 2011 - Hong Kong, China
    Duration: Feb 9 2011Feb 12 2011

    Publication series

    NameProceedings of the 4th ACM International Conference on Web Search and Data Mining, WSDM 2011

    Other

    Other4th ACM International Conference on Web Search and Data Mining, WSDM 2011
    CountryChina
    CityHong Kong
    Period2/9/112/12/11

    Keywords

    • Batch query processing
    • Query processing
    • Result cache updates
    • Web search

    ASJC Scopus subject areas

    • Computer Networks and Communications
    • Computer Science Applications
    • Software

    Fingerprint Dive into the research topics of 'Batch query processing for web search engines'. Together they form a unique fingerprint.

    Cite this