A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models

Tiwalayo Eisape, M. H. Tessler, Ishita Dasgupta, Fei Sha, Sjoerd van Steenkiste, Tal Linzen

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    A central component of rational behavior is logical inference: the process of determining which conclusions follow from a set of premises. Psychologists have documented several ways in which humans’ inferences deviate from the rules of logic. Do language models, which are trained on text generated by humans, replicate such human biases, or are they able to overcome them? Focusing on the case of syllogisms—inferences from two simple premises—we show that, within the PaLM 2 family of transformer language models, larger models are more logical than smaller ones, and also more logical than humans. At the same time, even the largest models make systematic errors, some of which mirror human reasoning biases: they show sensitivity to the (irrelevant) ordering of the variables in the syllogism, and draw confident but incorrect inferences from particular syllogisms (syllogistic fallacies). Overall, we find that language models often mimic the human biases included in their training data, but are able to overcome them in some cases.

    Original languageEnglish (US)
    Title of host publicationLong Papers
    EditorsKevin Duh, Helena Gomez, Steven Bethard
    PublisherAssociation for Computational Linguistics (ACL)
    Pages8418-8437
    Number of pages20
    ISBN (Electronic)9798891761148
    StatePublished - 2024
    Event2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024 - Hybrid, Mexico City, Mexico
    Duration: Jun 16 2024Jun 21 2024

    Publication series

    NameProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024
    Volume1

    Conference

    Conference2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024
    Country/TerritoryMexico
    CityHybrid, Mexico City
    Period6/16/246/21/24

    ASJC Scopus subject areas

    • Computer Networks and Communications
    • Hardware and Architecture
    • Information Systems
    • Software

    Fingerprint

    Dive into the research topics of 'A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models'. Together they form a unique fingerprint.

    Cite this