Using priming to uncover the organization of syntactic representations in neural language models

Grusha Prasad; Marten Van Schijndel; Tal Linzen

Using priming to uncover the organization of syntactic representations in neural language models

Grusha Prasad, Marten Van Schijndel, Tal Linzen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Neural language models (LMs) perform well on tasks that require sensitivity to syntactic structure. Drawing on the syntactic priming paradigm from psycholinguistics, we propose a novel technique to analyze the representations that enable such success. By establishing a gradient similarity metric between structures, this technique allows us to reconstruct the organization of the LMs' syntactic representational space. We use this technique to demonstrate that LSTM LMs' representations of different types of sentences with relative clauses are organized hierarchically in a linguistically interpretable manner, suggesting that the LMs track abstract properties of the sentence.

Original language	English (US)
Title of host publication	CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference
Publisher	Association for Computational Linguistics
Pages	66-76
Number of pages	11
ISBN (Electronic)	9781950737727
State	Published - 2019
Event	23rd Conference on Computational Natural Language Learning, CoNLL 2019 - Hong Kong, China Duration: Nov 3 2019 → Nov 4 2019

Publication series

Name	CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference

Conference

Conference	23rd Conference on Computational Natural Language Learning, CoNLL 2019
Country/Territory	China
City	Hong Kong
Period	11/3/19 → 11/4/19

ASJC Scopus subject areas

Computer Science Applications
Information Systems
Computational Theory and Mathematics

Cite this

Prasad, G., Van Schijndel, M., & Linzen, T. (2019). Using priming to uncover the organization of syntactic representations in neural language models. In CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference (pp. 66-76). (CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference). Association for Computational Linguistics.

Using priming to uncover the organization of syntactic representations in neural language models. / Prasad, Grusha; Van Schijndel, Marten; Linzen, Tal.

CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference. Association for Computational Linguistics, 2019. p. 66-76 (CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Prasad, G, Van Schijndel, M & Linzen, T 2019, Using priming to uncover the organization of syntactic representations in neural language models. in CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference. CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference, Association for Computational Linguistics, pp. 66-76, 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, 11/3/19.

Prasad G, Van Schijndel M, Linzen T. Using priming to uncover the organization of syntactic representations in neural language models. In CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference. Association for Computational Linguistics. 2019. p. 66-76. (CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference).

Prasad, Grusha ; Van Schijndel, Marten ; Linzen, Tal. / Using priming to uncover the organization of syntactic representations in neural language models. CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference. Association for Computational Linguistics, 2019. pp. 66-76 (CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference).

@inproceedings{54c25b337cfd4f68a53c8885fbbf849c,

title = "Using priming to uncover the organization of syntactic representations in neural language models",

abstract = "Neural language models (LMs) perform well on tasks that require sensitivity to syntactic structure. Drawing on the syntactic priming paradigm from psycholinguistics, we propose a novel technique to analyze the representations that enable such success. By establishing a gradient similarity metric between structures, this technique allows us to reconstruct the organization of the LMs' syntactic representational space. We use this technique to demonstrate that LSTM LMs' representations of different types of sentences with relative clauses are organized hierarchically in a linguistically interpretable manner, suggesting that the LMs track abstract properties of the sentence.",

author = "Grusha Prasad and {Van Schijndel}, Marten and Tal Linzen",

note = "Publisher Copyright: {\textcopyright} 2019 Association for Computational Linguistics. Copyright: Copyright 2020 Elsevier B.V., All rights reserved.; 23rd Conference on Computational Natural Language Learning, CoNLL 2019 ; Conference date: 03-11-2019 Through 04-11-2019",

year = "2019",

language = "English (US)",

series = "CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference",

publisher = "Association for Computational Linguistics",

pages = "66--76",

booktitle = "CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference",

}

TY - GEN

T1 - Using priming to uncover the organization of syntactic representations in neural language models

AU - Prasad, Grusha

AU - Van Schijndel, Marten

AU - Linzen, Tal

PY - 2019

Y1 - 2019

N2 - Neural language models (LMs) perform well on tasks that require sensitivity to syntactic structure. Drawing on the syntactic priming paradigm from psycholinguistics, we propose a novel technique to analyze the representations that enable such success. By establishing a gradient similarity metric between structures, this technique allows us to reconstruct the organization of the LMs' syntactic representational space. We use this technique to demonstrate that LSTM LMs' representations of different types of sentences with relative clauses are organized hierarchically in a linguistically interpretable manner, suggesting that the LMs track abstract properties of the sentence.

AB - Neural language models (LMs) perform well on tasks that require sensitivity to syntactic structure. Drawing on the syntactic priming paradigm from psycholinguistics, we propose a novel technique to analyze the representations that enable such success. By establishing a gradient similarity metric between structures, this technique allows us to reconstruct the organization of the LMs' syntactic representational space. We use this technique to demonstrate that LSTM LMs' representations of different types of sentences with relative clauses are organized hierarchically in a linguistically interpretable manner, suggesting that the LMs track abstract properties of the sentence.

UR - http://www.scopus.com/inward/record.url?scp=85084330807&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85084330807&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85084330807

T3 - CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference

SP - 66

EP - 76

BT - CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference

PB - Association for Computational Linguistics

T2 - 23rd Conference on Computational Natural Language Learning, CoNLL 2019

Y2 - 3 November 2019 through 4 November 2019

ER -

Using priming to uncover the organization of syntactic representations in neural language models

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this