DSDD: Domain-Specific Dataset Discovery on the Web

Haoxiang Zhang, Aécio Santos, Juliana Freire

Research output: Chapter in Book/Report/Conference proceedingConference contribution


With the push for transparency and open data, many datasets and data repositories are becoming available on the Web. This opens new opportunities for data-driven exploration, from empowering analysts to answer new questions and obtain insights to improving predictive models through data augmentation. But as datasets are spread over a plethora of Web sites, finding data that are relevant for a given task is difficult. In this paper, we take a first step towards the construction of domain-specific data lakes. We propose an end-to-end dataset discovery system, targeted at domain experts, which given a small set of keywords, automatically finds potentially relevant datasets on the Web. The system makes use of search engines to hop across Web sites, uses online learning to incrementally build a model to recognize sites that contain datasets, utilizes a set of discovery actions to broaden the search, and applies a multi-armed bandit based algorithm to balance the trade-offs of different discovery actions. We report the results of an extensive experimental evaluation over multiple domains, and demonstrate that our strategy is effective and outperforms state-of-the-art content discovery methods.

Original languageEnglish (US)
Title of host publicationCIKM 2021 - Proceedings of the 30th ACM International Conference on Information and Knowledge Management
PublisherAssociation for Computing Machinery
Number of pages10
ISBN (Electronic)9781450384469
StatePublished - Oct 26 2021
Event30th ACM International Conference on Information and Knowledge Management, CIKM 2021 - Virtual, Online, Australia
Duration: Nov 1 2021Nov 5 2021

Publication series

NameInternational Conference on Information and Knowledge Management, Proceedings


Conference30th ACM International Conference on Information and Knowledge Management, CIKM 2021
CityVirtual, Online


  • domain-specific dataset discovery
  • focused crawling
  • meta search
  • multi-armed bandit
  • online learning

ASJC Scopus subject areas

  • General Business, Management and Accounting
  • General Decision Sciences


Dive into the research topics of 'DSDD: Domain-Specific Dataset Discovery on the Web'. Together they form a unique fingerprint.

Cite this