On the Democratization of Machine Learning Pipelines

Alexandre Carqueja, Bruno Cabral, João Paulo Fernandes, Nuno Lourenço

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the increase of Machine Learning (ML) adoption throughout many industries, the need for efficient methods to continuously design, develop, and deploy ML models has also grown. To address this issue, several ML pipelines have emerged with the goal of creating development environments which facilitate the deployment, evaluation and maintenance. In this paper, we advocate that most existing pipelines are not well suited for the initial stages of ML development, due to their high setup and maintenance overheads. As such, we propose a lightweight Quick Machine Learning framework, QML, which is capable of reducing the setup overhead and operating in the low infrastructure environments that are most common-place in experimental ML projects. To demonstrate QML's usefulness, we present a case-study where a lightweight ML pipeline was developed, and subsequently validated on a standard ML classification problem. Lastly, we assess the differences between our pipeline and an alternative lightweight workflow, based on DAGsHub. With this comparison, we conclude that our approach increases ML task automation as well as feature support, while falling short only in the Experiment Tracking category. To enable the broader community to experiment and assess QML, as well as the Lightweight Pipeline, this project has been made publicly available. https://github.com/WALEX2000/qml1 https://github.com/WALEX2000/qml1 https://github.com/WALEX2000/qml1 https://github.com/WALEX2000/qml.

Original languageEnglish (US)
Title of host publicationProceedings of the 2022 IEEE Symposium Series on Computational Intelligence, SSCI 2022
EditorsHisao Ishibuchi, Chee-Keong Kwoh, Ah-Hwee Tan, Dipti Srinivasan, Chunyan Miao, Anupam Trivedi, Keeley Crockett
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages455-462
Number of pages8
ISBN (Electronic)9781665487689
DOIs
StatePublished - 2022
Event2022 IEEE Symposium Series on Computational Intelligence, SSCI 2022 - Singapore, Singapore
Duration: Dec 4 2022Dec 7 2022

Publication series

NameProceedings of the 2022 IEEE Symposium Series on Computational Intelligence, SSCI 2022

Conference

Conference2022 IEEE Symposium Series on Computational Intelligence, SSCI 2022
Country/TerritorySingapore
CitySingapore
Period12/4/2212/7/22

Keywords

  • Lightweight
  • Machine Learning Pipelines
  • MLOps
  • Workflows

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Decision Sciences (miscellaneous)
  • Computational Mathematics
  • Control and Optimization
  • Transportation

Fingerprint

Dive into the research topics of 'On the Democratization of Machine Learning Pipelines'. Together they form a unique fingerprint.

Cite this