A fast and robust method for web page template detection and removal

Karane Vieira, Altigran S. Da Silva, Nick Pinto, Edleno S. De Moura, Joo M B Cavalcanti, Juliana Freire

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The widespread use of templates on the Web is considered harmful for two main reasons. Not only do they compromise the relevance judgment of many web IR and web mining methods such as clustering and classification, but they also negatively impact the performance and resource usage of tools that process web pages. In this paper we present a new method that efficiently and accurately removes templates found in collections of web pages. Our method works in two steps. First, the costly process of template detection is performed over a small set of sample pages. Then, the derived template is removed from the remaining pages in the collection. This leads to substantial performance gains when compared to previous approaches that combine template detection and removal. We show, through an experimental evaluation, that our approach is effective for identifying terms occurring in templates - obtaining F-measure values around 0.9, and that it also boosts the accuracy of web page clustering and classification methods.

Original languageEnglish (US)
Title of host publicationProceedings of the 15th ACM Conference on Information and Knowledge Management, CIKM 2006
Pages258-267
Number of pages10
DOIs
StatePublished - 2006
Event15th ACM Conference on Information and Knowledge Management, CIKM 2006 - Arlington, VA, United States
Duration: Nov 6 2006Nov 11 2006

Publication series

NameInternational Conference on Information and Knowledge Management, Proceedings

Other

Other15th ACM Conference on Information and Knowledge Management, CIKM 2006
CountryUnited States
CityArlington, VA
Period11/6/0611/11/06

Keywords

  • Web page noise removal
  • Web template extraction

ASJC Scopus subject areas

  • Decision Sciences(all)
  • Business, Management and Accounting(all)

Fingerprint Dive into the research topics of 'A fast and robust method for web page template detection and removal'. Together they form a unique fingerprint.

Cite this