How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN

R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, Asli Celikyilmaz

    Research output: Contribution to journalArticlepeer-review

    Abstract

    Current language models can generate highquality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? To tease apart these possibilities, we introduce RAVEN, a suite of analyses for assessing the novelty of generated text, focusing on sequential structure (n-grams) and syntactic structure. We apply these analyses to four neural language models trained on English (an LSTM, a Transformer, Transformer-XL, and GPT-2). For local structure—e.g., individual dependencies— text generated with a standard sampling scheme is substantially less novel than our baseline of human-generated text from each model’s test set. For larger-scale structure— e.g., overall sentence structure—modelgenerated text is as novel or even more novel than the human-generated baseline, but models still sometimes copy substantially, in some cases duplicating passages over 1,000 words long from the training set. We also perform extensive manual analysis, finding evidence that GPT-2 uses both compositional and analogical generalization mechanisms and showing that GPT-2’s novel text is usuallywell-formed morphologically and syntactically but has reasonably frequent semantic issues (e.g., being self-contradictory).

    Original languageEnglish (US)
    Pages (from-to)652-670
    Number of pages19
    JournalTransactions of the Association for Computational Linguistics
    Volume11
    DOIs
    StatePublished - Jun 29 2023

    ASJC Scopus subject areas

    • Communication
    • Human-Computer Interaction
    • Linguistics and Language
    • Computer Science Applications
    • Artificial Intelligence

    Fingerprint

    Dive into the research topics of 'How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN'. Together they form a unique fingerprint.

    Cite this