Exploring Size-Speed Trade-Offs in Static Index Pruning

Juan Rodriguez; Torsten Suel

doi:10.1109/BigData.2018.8622177

Exploring Size-Speed Trade-Offs in Static Index Pruning

Juan Rodriguez, Torsten Suel

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Static index pruning techniques remove postings from inverted index structures in order to decrease index size and query processing cost, while minimizing the resulting loss in result quality. A number of authors have proposed pruning techniques that use basic properties of postings as well as results of past queries to decide what postings should be kept. However, many open questions remain, and our goal is to address some of them using a machine learning based approach that tries to predict the usefulness of a posting. In this paper, we explore the following questions: (1) How much does an approach that learns from a rich set of features outperform previous work that uses heuristic approaches or just a few features? (2) What is the relationship between index size and query processing speed in static index pruning? We show that an approach that prunes postings using a rich set of features including post-hits and doc-hits can significantly outperform previous approaches, and that there is a very pronounced trade-off between index size and query processing speed for static index pruning that has not been previously explored.

Original language	English (US)
Title of host publication	Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018
Editors	Naoki Abe, Huan Liu, Calton Pu, Xiaohua Hu, Nesreen Ahmed, Mu Qiao, Yang Song, Donald Kossmann, Bing Liu, Kisung Lee, Jiliang Tang, Jingrui He, Jeffrey Saltz
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1093-1100
Number of pages	8
ISBN (Electronic)	9781538650356
DOIs	https://doi.org/10.1109/BigData.2018.8622177
State	Published - Jul 2 2018
Event	2018 IEEE International Conference on Big Data, Big Data 2018 - Seattle, United States Duration: Dec 10 2018 → Dec 13 2018

Publication series

Name	Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

Conference

Conference	2018 IEEE International Conference on Big Data, Big Data 2018
Country/Territory	United States
City	Seattle
Period	12/10/18 → 12/13/18

Keywords

search engine performance
search optimization
static index pruning
web search engine

ASJC Scopus subject areas

Computer Science Applications
Information Systems

Access to Document

10.1109/BigData.2018.8622177

Cite this

Rodriguez, J., & Suel, T. (2018). Exploring Size-Speed Trade-Offs in Static Index Pruning. In N. Abe, H. Liu, C. Pu, X. Hu, N. Ahmed, M. Qiao, Y. Song, D. Kossmann, B. Liu, K. Lee, J. Tang, J. He, & J. Saltz (Eds.), Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018 (pp. 1093-1100). Article 8622177 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData.2018.8622177

Exploring Size-Speed Trade-Offs in Static Index Pruning. / Rodriguez, Juan; Suel, Torsten.
Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. ed. / Naoki Abe; Huan Liu; Calton Pu; Xiaohua Hu; Nesreen Ahmed; Mu Qiao; Yang Song; Donald Kossmann; Bing Liu; Kisung Lee; Jiliang Tang; Jingrui He; Jeffrey Saltz. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1093-1100 8622177 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Rodriguez, J & Suel, T 2018, Exploring Size-Speed Trade-Offs in Static Index Pruning. in N Abe, H Liu, C Pu, X Hu, N Ahmed, M Qiao, Y Song, D Kossmann, B Liu, K Lee, J Tang, J He & J Saltz (eds), Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018., 8622177, Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018, Institute of Electrical and Electronics Engineers Inc., pp. 1093-1100, 2018 IEEE International Conference on Big Data, Big Data 2018, Seattle, United States, 12/10/18. https://doi.org/10.1109/BigData.2018.8622177

Rodriguez J, Suel T. Exploring Size-Speed Trade-Offs in Static Index Pruning. In Abe N, Liu H, Pu C, Hu X, Ahmed N, Qiao M, Song Y, Kossmann D, Liu B, Lee K, Tang J, He J, Saltz J, editors, Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1093-1100. 8622177. (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018). doi: 10.1109/BigData.2018.8622177

Rodriguez, Juan ; Suel, Torsten. / Exploring Size-Speed Trade-Offs in Static Index Pruning. Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. editor / Naoki Abe ; Huan Liu ; Calton Pu ; Xiaohua Hu ; Nesreen Ahmed ; Mu Qiao ; Yang Song ; Donald Kossmann ; Bing Liu ; Kisung Lee ; Jiliang Tang ; Jingrui He ; Jeffrey Saltz. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 1093-1100 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018).

@inproceedings{6ec6d39876514110bf61be2b7b5de2f5,

title = "Exploring Size-Speed Trade-Offs in Static Index Pruning",

abstract = "Static index pruning techniques remove postings from inverted index structures in order to decrease index size and query processing cost, while minimizing the resulting loss in result quality. A number of authors have proposed pruning techniques that use basic properties of postings as well as results of past queries to decide what postings should be kept. However, many open questions remain, and our goal is to address some of them using a machine learning based approach that tries to predict the usefulness of a posting. In this paper, we explore the following questions: (1) How much does an approach that learns from a rich set of features outperform previous work that uses heuristic approaches or just a few features? (2) What is the relationship between index size and query processing speed in static index pruning? We show that an approach that prunes postings using a rich set of features including post-hits and doc-hits can significantly outperform previous approaches, and that there is a very pronounced trade-off between index size and query processing speed for static index pruning that has not been previously explored.",

keywords = "search engine performance, search optimization, static index pruning, web search engine",

author = "Juan Rodriguez and Torsten Suel",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 2018 IEEE International Conference on Big Data, Big Data 2018 ; Conference date: 10-12-2018 Through 13-12-2018",

year = "2018",

month = jul,

day = "2",

doi = "10.1109/BigData.2018.8622177",

language = "English (US)",

series = "Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1093--1100",

editor = "Naoki Abe and Huan Liu and Calton Pu and Xiaohua Hu and Nesreen Ahmed and Mu Qiao and Yang Song and Donald Kossmann and Bing Liu and Kisung Lee and Jiliang Tang and Jingrui He and Jeffrey Saltz",

booktitle = "Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018",

}

TY - GEN

T1 - Exploring Size-Speed Trade-Offs in Static Index Pruning

AU - Rodriguez, Juan

AU - Suel, Torsten

PY - 2018/7/2

Y1 - 2018/7/2

N2 - Static index pruning techniques remove postings from inverted index structures in order to decrease index size and query processing cost, while minimizing the resulting loss in result quality. A number of authors have proposed pruning techniques that use basic properties of postings as well as results of past queries to decide what postings should be kept. However, many open questions remain, and our goal is to address some of them using a machine learning based approach that tries to predict the usefulness of a posting. In this paper, we explore the following questions: (1) How much does an approach that learns from a rich set of features outperform previous work that uses heuristic approaches or just a few features? (2) What is the relationship between index size and query processing speed in static index pruning? We show that an approach that prunes postings using a rich set of features including post-hits and doc-hits can significantly outperform previous approaches, and that there is a very pronounced trade-off between index size and query processing speed for static index pruning that has not been previously explored.

AB - Static index pruning techniques remove postings from inverted index structures in order to decrease index size and query processing cost, while minimizing the resulting loss in result quality. A number of authors have proposed pruning techniques that use basic properties of postings as well as results of past queries to decide what postings should be kept. However, many open questions remain, and our goal is to address some of them using a machine learning based approach that tries to predict the usefulness of a posting. In this paper, we explore the following questions: (1) How much does an approach that learns from a rich set of features outperform previous work that uses heuristic approaches or just a few features? (2) What is the relationship between index size and query processing speed in static index pruning? We show that an approach that prunes postings using a rich set of features including post-hits and doc-hits can significantly outperform previous approaches, and that there is a very pronounced trade-off between index size and query processing speed for static index pruning that has not been previously explored.

KW - search engine performance

KW - search optimization

KW - static index pruning

KW - web search engine

UR - http://www.scopus.com/inward/record.url?scp=85062624645&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062624645&partnerID=8YFLogxK

U2 - 10.1109/BigData.2018.8622177

DO - 10.1109/BigData.2018.8622177

M3 - Conference contribution

AN - SCOPUS:85062624645

T3 - Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

SP - 1093

EP - 1100

BT - Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

A2 - Abe, Naoki

A2 - Liu, Huan

A2 - Pu, Calton

A2 - Hu, Xiaohua

A2 - Ahmed, Nesreen

A2 - Qiao, Mu

A2 - Song, Yang

A2 - Kossmann, Donald

A2 - Liu, Bing

A2 - Lee, Kisung

A2 - Tang, Jiliang

A2 - He, Jingrui

A2 - Saltz, Jeffrey

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2018 IEEE International Conference on Big Data, Big Data 2018

Y2 - 10 December 2018 through 13 December 2018

ER -

Exploring Size-Speed Trade-Offs in Static Index Pruning

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this