TY - GEN
T1 - Comparing character-level neural language models using a lexical decision task
AU - Le Godais, Gaël
AU - Linzen, Tal
AU - Dupoux, Emmanuel
N1 - Publisher Copyright:
© 2017 Association for Computational Linguistics.
PY - 2017
Y1 - 2017
N2 - What is the information captured by neural network models of language? We address this question in the case of character-level recurrent neural language models. These models do not have explicit word representations; do they acquire implicit ones? We assess the lexical capacity of a network using the lexical decision task common in psycholinguistics: the system is required to decide whether or not a string of characters forms a word. We explore how accuracy on this task is affected by the architecture of the network, focusing on cell type (LSTM vs. SRN), depth and width. We also compare these architectural properties to a simple count of the parameters of the network. The overall number of parameters in the network turns out to be the most important predictor of accuracy; in particular, there is little evidence that deeper networks are beneficial for this task.
AB - What is the information captured by neural network models of language? We address this question in the case of character-level recurrent neural language models. These models do not have explicit word representations; do they acquire implicit ones? We assess the lexical capacity of a network using the lexical decision task common in psycholinguistics: the system is required to decide whether or not a string of characters forms a word. We explore how accuracy on this task is affected by the architecture of the network, focusing on cell type (LSTM vs. SRN), depth and width. We also compare these architectural properties to a simple count of the parameters of the network. The overall number of parameters in the network turns out to be the most important predictor of accuracy; in particular, there is little evidence that deeper networks are beneficial for this task.
UR - http://www.scopus.com/inward/record.url?scp=85021647927&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85021647927&partnerID=8YFLogxK
U2 - 10.18653/v1/e17-2020
DO - 10.18653/v1/e17-2020
M3 - Conference contribution
AN - SCOPUS:85021647927
T3 - 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017 - Proceedings of Conference
SP - 125
EP - 130
BT - Short Papers
PB - Association for Computational Linguistics (ACL)
T2 - 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017
Y2 - 3 April 2017 through 7 April 2017
ER -