Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

Julia Koehler Leman; Sergey Lyskov; Steven M. Lewis; Jared Adolf-Bryfogle; Rebecca F. Alford; Kyle Barlow; Ziv Ben-Aharon; Daniel Farrell; Jason Fell; William A. Hansen; Ameya Harmalkar; Jeliazko Jeliazkov; Georg Kuenze; Justyna D. Krys; Ajasja Ljubetič; Amanda L. Loshbaugh; Jack Maguire; Rocco Moretti; Vikram Khipple Mulligan; Morgan L. Nance; Phuong T. Nguyen; Shane Ó Conchúir; Shourya S. Roy Burman; Rituparna Samanta; Shannon T. Smith; Frank Teets; Johanna K.S. Tiemann; Andrew Watkins; Hope Woods; Brahm J. Yachnin; Christopher D. Bahl; Chris Bailey-Kellogg; David Baker; Rhiju Das; Frank DiMaio; Sagar D. Khare; Tanja Kortemme; Jason W. Labonte; Kresten Lindorff-Larsen; Jens Meiler; William Schief; Ora Schueler-Furman; Justin B. Siegel; Amelie Stein; Vladimir Yarov-Yarovoy; Brian Kuhlman; Andrew Leaver-Fay; Dominik Gront; Jeffrey J. Gray; Richard Bonneau

doi:10.1038/s41467-021-27222-7

Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

Julia Koehler Leman, Sergey Lyskov, Steven M. Lewis, Jared Adolf-Bryfogle, Rebecca F. Alford, Kyle Barlow, Ziv Ben-Aharon, Daniel Farrell, Jason Fell, William A. Hansen, Ameya Harmalkar, Jeliazko Jeliazkov, Georg Kuenze, Justyna D. Krys, Ajasja Ljubetič, Amanda L. Loshbaugh, Jack Maguire, Rocco Moretti, Vikram Khipple Mulligan, Morgan L. NancePhuong T. Nguyen, Shane Ó Conchúir, Shourya S. Roy Burman, Rituparna Samanta, Shannon T. Smith, Frank Teets, Johanna K.S. Tiemann, Andrew Watkins, Hope Woods, Brahm J. Yachnin, Christopher D. Bahl, Chris Bailey-Kellogg, David Baker, Rhiju Das, Frank DiMaio, Sagar D. Khare, Tanja Kortemme, Jason W. Labonte, Kresten Lindorff-Larsen, Jens Meiler, William Schief, Ora Schueler-Furman, Justin B. Siegel, Amelie Stein, Vladimir Yarov-Yarovoy, Brian Kuhlman, Andrew Leaver-Fay, Dominik Gront, Jeffrey J. Gray, Richard Bonneau

Research output: Contribution to journal › Article › peer-review

Abstract

Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.

Original language	English (US)
Article number	6947
Journal	Nature communications
Volume	12
Issue number	1
DOIs	https://doi.org/10.1038/s41467-021-27222-7
State	Published - Dec 2021

ASJC Scopus subject areas

General Chemistry
General Biochemistry, Genetics and Molecular Biology
General Physics and Astronomy

Access to Document

10.1038/s41467-021-27222-7

Cite this

Koehler Leman, J., Lyskov, S., Lewis, S. M., Adolf-Bryfogle, J., Alford, R. F., Barlow, K., Ben-Aharon, Z., Farrell, D., Fell, J., Hansen, W. A., Harmalkar, A., Jeliazkov, J., Kuenze, G., Krys, J. D., Ljubetič, A., Loshbaugh, A. L., Maguire, J., Moretti, R., Mulligan, V. K., ... Bonneau, R. (2021). Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks. Nature communications, 12(1), Article 6947. https://doi.org/10.1038/s41467-021-27222-7

Koehler Leman, J, Lyskov, S, Lewis, SM, Adolf-Bryfogle, J, Alford, RF, Barlow, K, Ben-Aharon, Z, Farrell, D, Fell, J, Hansen, WA, Harmalkar, A, Jeliazkov, J, Kuenze, G, Krys, JD, Ljubetič, A, Loshbaugh, AL, Maguire, J, Moretti, R, Mulligan, VK, Nance, ML, Nguyen, PT, Ó Conchúir, S, Roy Burman, SS, Samanta, R, Smith, ST, Teets, F, Tiemann, JKS, Watkins, A, Woods, H, Yachnin, BJ, Bahl, CD, Bailey-Kellogg, C, Baker, D, Das, R, DiMaio, F, Khare, SD, Kortemme, T, Labonte, JW, Lindorff-Larsen, K, Meiler, J, Schief, W, Schueler-Furman, O, Siegel, JB, Stein, A, Yarov-Yarovoy, V, Kuhlman, B, Leaver-Fay, A, Gront, D, Gray, JJ & Bonneau, R 2021, 'Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks', Nature communications, vol. 12, no. 1, 6947. https://doi.org/10.1038/s41467-021-27222-7

@article{4bbd62ff0dae485c9c300239ff16ac8a,

title = "Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks",

abstract = "Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.",

author = "{Koehler Leman}, Julia and Sergey Lyskov and Lewis, {Steven M.} and Jared Adolf-Bryfogle and Alford, {Rebecca F.} and Kyle Barlow and Ziv Ben-Aharon and Daniel Farrell and Jason Fell and Hansen, {William A.} and Ameya Harmalkar and Jeliazko Jeliazkov and Georg Kuenze and Krys, {Justyna D.} and Ajasja Ljubeti{\v c} and Loshbaugh, {Amanda L.} and Jack Maguire and Rocco Moretti and Mulligan, {Vikram Khipple} and Nance, {Morgan L.} and Nguyen, {Phuong T.} and {{\'O} Conch{\'u}ir}, Shane and {Roy Burman}, {Shourya S.} and Rituparna Samanta and Smith, {Shannon T.} and Frank Teets and Tiemann, {Johanna K.S.} and Andrew Watkins and Hope Woods and Yachnin, {Brahm J.} and Bahl, {Christopher D.} and Chris Bailey-Kellogg and David Baker and Rhiju Das and Frank DiMaio and Khare, {Sagar D.} and Tanja Kortemme and Labonte, {Jason W.} and Kresten Lindorff-Larsen and Jens Meiler and William Schief and Ora Schueler-Furman and Siegel, {Justin B.} and Amelie Stein and Vladimir Yarov-Yarovoy and Brian Kuhlman and Andrew Leaver-Fay and Dominik Gront and Gray, {Jeffrey J.} and Richard Bonneau",

note = "Publisher Copyright: {\textcopyright} 2021, The Author(s).",

year = "2021",

month = dec,

doi = "10.1038/s41467-021-27222-7",

language = "English (US)",

volume = "12",

journal = "Nature communications",

issn = "2041-1723",

publisher = "Nature Research",

number = "1",

}

TY - JOUR

T1 - Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

AU - Koehler Leman, Julia

AU - Lyskov, Sergey

AU - Lewis, Steven M.

AU - Adolf-Bryfogle, Jared

AU - Alford, Rebecca F.

AU - Barlow, Kyle

AU - Ben-Aharon, Ziv

AU - Farrell, Daniel

AU - Fell, Jason

AU - Hansen, William A.

AU - Harmalkar, Ameya

AU - Jeliazkov, Jeliazko

AU - Kuenze, Georg

AU - Krys, Justyna D.

AU - Ljubetič, Ajasja

AU - Loshbaugh, Amanda L.

AU - Maguire, Jack

AU - Moretti, Rocco

AU - Mulligan, Vikram Khipple

AU - Nance, Morgan L.

AU - Nguyen, Phuong T.

AU - Ó Conchúir, Shane

AU - Roy Burman, Shourya S.

AU - Samanta, Rituparna

AU - Smith, Shannon T.

AU - Teets, Frank

AU - Tiemann, Johanna K.S.

AU - Watkins, Andrew

AU - Woods, Hope

AU - Yachnin, Brahm J.

AU - Bahl, Christopher D.

AU - Bailey-Kellogg, Chris

AU - Baker, David

AU - Das, Rhiju

AU - DiMaio, Frank

AU - Khare, Sagar D.

AU - Kortemme, Tanja

AU - Labonte, Jason W.

AU - Lindorff-Larsen, Kresten

AU - Meiler, Jens

AU - Schief, William

AU - Schueler-Furman, Ora

AU - Siegel, Justin B.

AU - Stein, Amelie

AU - Yarov-Yarovoy, Vladimir

AU - Kuhlman, Brian

AU - Leaver-Fay, Andrew

AU - Gront, Dominik

AU - Gray, Jeffrey J.

AU - Bonneau, Richard

PY - 2021/12

Y1 - 2021/12

N2 - Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.

AB - Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.

UR - http://www.scopus.com/inward/record.url?scp=85120055073&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85120055073&partnerID=8YFLogxK

U2 - 10.1038/s41467-021-27222-7

DO - 10.1038/s41467-021-27222-7

M3 - Article

C2 - 34845212

AN - SCOPUS:85120055073

SN - 2041-1723

VL - 12

JO - Nature communications

JF - Nature communications

IS - 1

M1 - 6947

ER -

Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

Abstract

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this