Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D'Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, Ellie Pavlick

    Research output: Contribution to conferencePaperpeer-review


    Experiments with pre-trained models such as BERT are often based on a single checkpoint. While the conclusions drawn apply to the artifact tested in the experiment (i.e., the particular instance of the model), it is not always clear whether they hold for the more general procedure which includes the architecture, training data, initialization scheme, and loss function. Recent work has shown that repeating the pre-training process can lead to substantially different performance, suggesting that an alternate strategy is needed to make principled statements about procedures. To enable researchers to draw more robust conclusions, we introduce the MultiBERTs, a set of 25 BERT-Base checkpoints, trained with similar hyper-parameters as the original BERT model but differing in random weight initialization and shuffling of training data. We also define the Multi-Bootstrap, a non-parametric bootstrap method for statistical inference designed for settings where there are multiple pre-trained models and limited test data. To illustrate our approach, we present a case study of gender bias in coreference resolution, in which the Multi-Bootstrap lets us measure effects that may not be detected with a single checkpoint. We release our models and statistical library, along with an additional set of 140 intermediate checkpoints captured during pre-training to facilitate research on learning dynamics.

    Original languageEnglish (US)
    StatePublished - 2022
    Event10th International Conference on Learning Representations, ICLR 2022 - Virtual, Online
    Duration: Apr 25 2022Apr 29 2022


    Conference10th International Conference on Learning Representations, ICLR 2022
    CityVirtual, Online

    ASJC Scopus subject areas

    • Language and Linguistics
    • Computer Science Applications
    • Education
    • Linguistics and Language


    Dive into the research topics of 'THE MULTIBERTS: BERT REPRODUCTIONS FOR ROBUSTNESS ANALYSIS'. Together they form a unique fingerprint.

    Cite this