TY - GEN
T1 - Batch query processing for web search engines
AU - Ding, Shuai
AU - Baeza-Yates, Ricardo
AU - Attenberg, Josh
AU - Suel, Torsten
PY - 2011
Y1 - 2011
N2 - Large web search engines are now processing billions of queries per day. Most of these queries are interactive in nature, requiring a response in fractions of a second. However, there are also a number of important scenarios where large batches of queries are submitted for various web mining and system optimization tasks that do not require an immediate response. Given the significant cost of executing search queries over billions of web pages, it is a natural question to ask if such batches of queries can be more efficiently executed than interactive queries. In this paper, we motivate and discuss the problem of batch query processing in search engines, identify basic mechanisms for improving the performance of such queries, and provide a preliminary experimental evaluation of the proposed techniques. Our conclusion is that significant cost reductions are possible by using specialized mechanisms for executing batch queries in Web search engines.
AB - Large web search engines are now processing billions of queries per day. Most of these queries are interactive in nature, requiring a response in fractions of a second. However, there are also a number of important scenarios where large batches of queries are submitted for various web mining and system optimization tasks that do not require an immediate response. Given the significant cost of executing search queries over billions of web pages, it is a natural question to ask if such batches of queries can be more efficiently executed than interactive queries. In this paper, we motivate and discuss the problem of batch query processing in search engines, identify basic mechanisms for improving the performance of such queries, and provide a preliminary experimental evaluation of the proposed techniques. Our conclusion is that significant cost reductions are possible by using specialized mechanisms for executing batch queries in Web search engines.
KW - Batch query processing
KW - Query processing
KW - Result cache updates
KW - Web search
UR - http://www.scopus.com/inward/record.url?scp=79952384748&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=79952384748&partnerID=8YFLogxK
U2 - 10.1145/1935826.1935858
DO - 10.1145/1935826.1935858
M3 - Conference contribution
AN - SCOPUS:79952384748
SN - 9781450304931
T3 - Proceedings of the 4th ACM International Conference on Web Search and Data Mining, WSDM 2011
SP - 137
EP - 146
BT - Proceedings of the 4th ACM International Conference on Web Search and Data Mining, WSDM 2011
T2 - 4th ACM International Conference on Web Search and Data Mining, WSDM 2011
Y2 - 9 February 2011 through 12 February 2011
ER -