A large-scale study about quality and reproducibility of jupyter notebooks

Joao Felipe Pimentel; Leonardo Murta; Vanessa Braganholo; Juliana Freire

doi:10.1109/MSR.2019.00077

A large-scale study about quality and reproducibility of jupyter notebooks

Joao Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, Juliana Freire

Urban Initiative

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Jupyter Notebooks have been widely adopted by many different communities, both in science and industry. They support the creation of literate programming documents that combine code, text, and execution results with visualizations and all sorts of rich media. The self-documenting aspects and the ability to reproduce results have been touted as significant benefits of notebooks. At the same time, there has been growing criticism that the way notebooks are being used leads to unexpected behavior, encourage poor coding practices, and that their results can be hard to reproduce. To understand good and bad practices used in the development of real notebooks, we studied 1.4 million notebooks from GitHub. We present a detailed analysis of their characteristics that impact reproducibility. We also propose a set of best practices that can improve the rate of reproducibility and discuss open challenges that require further research and development.

Original language	English (US)
Title of host publication	Proceedings - 2019 IEEE/ACM 16th International Conference on Mining Software Repositories, MSR 2019
Publisher	IEEE Computer Society
Pages	507-517
Number of pages	11
ISBN (Electronic)	9781728134123
DOIs	https://doi.org/10.1109/MSR.2019.00077
State	Published - May 2019
Event	16th IEEE/ACM International Conference on Mining Software Repositories, MSR 2019 - Montreal, Canada Duration: May 26 2019 → May 27 2019

Publication series

Name	IEEE International Working Conference on Mining Software Repositories
Volume	2019-May
ISSN (Print)	2160-1852
ISSN (Electronic)	2160-1860

Conference

Conference	16th IEEE/ACM International Conference on Mining Software Repositories, MSR 2019
Country/Territory	Canada
City	Montreal
Period	5/26/19 → 5/27/19

Keywords

Github
Jupyter notebook
Reproducibility

ASJC Scopus subject areas

Computer Science Applications
Software

Access to Document

10.1109/MSR.2019.00077

Cite this

Pimentel, J. F., Murta, L., Braganholo, V., & Freire, J. (2019). A large-scale study about quality and reproducibility of jupyter notebooks. In Proceedings - 2019 IEEE/ACM 16th International Conference on Mining Software Repositories, MSR 2019 (pp. 507-517). Article 8816763 (IEEE International Working Conference on Mining Software Repositories; Vol. 2019-May). IEEE Computer Society. https://doi.org/10.1109/MSR.2019.00077

A large-scale study about quality and reproducibility of jupyter notebooks. / Pimentel, Joao Felipe; Murta, Leonardo; Braganholo, Vanessa et al.
Proceedings - 2019 IEEE/ACM 16th International Conference on Mining Software Repositories, MSR 2019. IEEE Computer Society, 2019. p. 507-517 8816763 (IEEE International Working Conference on Mining Software Repositories; Vol. 2019-May).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Pimentel, JF, Murta, L, Braganholo, V & Freire, J 2019, A large-scale study about quality and reproducibility of jupyter notebooks. in Proceedings - 2019 IEEE/ACM 16th International Conference on Mining Software Repositories, MSR 2019., 8816763, IEEE International Working Conference on Mining Software Repositories, vol. 2019-May, IEEE Computer Society, pp. 507-517, 16th IEEE/ACM International Conference on Mining Software Repositories, MSR 2019, Montreal, Canada, 5/26/19. https://doi.org/10.1109/MSR.2019.00077

Pimentel JF, Murta L, Braganholo V, Freire J. A large-scale study about quality and reproducibility of jupyter notebooks. In Proceedings - 2019 IEEE/ACM 16th International Conference on Mining Software Repositories, MSR 2019. IEEE Computer Society. 2019. p. 507-517. 8816763. (IEEE International Working Conference on Mining Software Repositories). doi: 10.1109/MSR.2019.00077

@inproceedings{43ca42dbba3b4b8c9802c067c69c6043,

title = "A large-scale study about quality and reproducibility of jupyter notebooks",

abstract = "Jupyter Notebooks have been widely adopted by many different communities, both in science and industry. They support the creation of literate programming documents that combine code, text, and execution results with visualizations and all sorts of rich media. The self-documenting aspects and the ability to reproduce results have been touted as significant benefits of notebooks. At the same time, there has been growing criticism that the way notebooks are being used leads to unexpected behavior, encourage poor coding practices, and that their results can be hard to reproduce. To understand good and bad practices used in the development of real notebooks, we studied 1.4 million notebooks from GitHub. We present a detailed analysis of their characteristics that impact reproducibility. We also propose a set of best practices that can improve the rate of reproducibility and discuss open challenges that require further research and development.",

keywords = "Github, Jupyter notebook, Reproducibility",

author = "Pimentel, {Joao Felipe} and Leonardo Murta and Vanessa Braganholo and Juliana Freire",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 16th IEEE/ACM International Conference on Mining Software Repositories, MSR 2019 ; Conference date: 26-05-2019 Through 27-05-2019",

year = "2019",

month = may,

doi = "10.1109/MSR.2019.00077",

language = "English (US)",

series = "IEEE International Working Conference on Mining Software Repositories",

publisher = "IEEE Computer Society",

pages = "507--517",

booktitle = "Proceedings - 2019 IEEE/ACM 16th International Conference on Mining Software Repositories, MSR 2019",

}

TY - GEN

T1 - A large-scale study about quality and reproducibility of jupyter notebooks

AU - Pimentel, Joao Felipe

AU - Murta, Leonardo

AU - Braganholo, Vanessa

AU - Freire, Juliana

PY - 2019/5

Y1 - 2019/5

N2 - Jupyter Notebooks have been widely adopted by many different communities, both in science and industry. They support the creation of literate programming documents that combine code, text, and execution results with visualizations and all sorts of rich media. The self-documenting aspects and the ability to reproduce results have been touted as significant benefits of notebooks. At the same time, there has been growing criticism that the way notebooks are being used leads to unexpected behavior, encourage poor coding practices, and that their results can be hard to reproduce. To understand good and bad practices used in the development of real notebooks, we studied 1.4 million notebooks from GitHub. We present a detailed analysis of their characteristics that impact reproducibility. We also propose a set of best practices that can improve the rate of reproducibility and discuss open challenges that require further research and development.

AB - Jupyter Notebooks have been widely adopted by many different communities, both in science and industry. They support the creation of literate programming documents that combine code, text, and execution results with visualizations and all sorts of rich media. The self-documenting aspects and the ability to reproduce results have been touted as significant benefits of notebooks. At the same time, there has been growing criticism that the way notebooks are being used leads to unexpected behavior, encourage poor coding practices, and that their results can be hard to reproduce. To understand good and bad practices used in the development of real notebooks, we studied 1.4 million notebooks from GitHub. We present a detailed analysis of their characteristics that impact reproducibility. We also propose a set of best practices that can improve the rate of reproducibility and discuss open challenges that require further research and development.

KW - Github

KW - Jupyter notebook

KW - Reproducibility

UR - http://www.scopus.com/inward/record.url?scp=85072330312&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072330312&partnerID=8YFLogxK

U2 - 10.1109/MSR.2019.00077

DO - 10.1109/MSR.2019.00077

M3 - Conference contribution

AN - SCOPUS:85072330312

T3 - IEEE International Working Conference on Mining Software Repositories

SP - 507

EP - 517

BT - Proceedings - 2019 IEEE/ACM 16th International Conference on Mining Software Repositories, MSR 2019

PB - IEEE Computer Society

T2 - 16th IEEE/ACM International Conference on Mining Software Repositories, MSR 2019

Y2 - 26 May 2019 through 27 May 2019

ER -

A large-scale study about quality and reproducibility of jupyter notebooks

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this