Issues in evaluating semantic spaces using word analogies

Tal Linzen

doi:10.18653/v1/w16-2503

Issues in evaluating semantic spaces using word analogies

Tal Linzen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: It is considered desirable for a space to represent semantic relations as consistent vector offsets. We show that the method's reliance on cosine similarity conflates offset consistency with largely irrelevant neighborhood structure, and propose simple baselines that should be used to improve the utility of the method in vector space evaluation.

Original language	English (US)
Title of host publication	Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016
Publisher	Association for Computational Linguistics (ACL)
Pages	13-18
Number of pages	6
ISBN (Electronic)	9781945626142
DOIs	https://doi.org/10.18653/v1/w16-2503
State	Published - 2016
Event	1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Berlin, Germany Duration: Aug 7 2016 → …

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016
Country/Territory	Germany
City	Berlin
Period	8/7/16 → …

ASJC Scopus subject areas

Computer Science Applications
Linguistics and Language
Language and Linguistics

Access to Document

10.18653/v1/w16-2503

Cite this

Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 (pp. 13-18). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/w16-2503

Issues in evaluating semantic spaces using word analogies. / Linzen, Tal.
Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. Association for Computational Linguistics (ACL), 2016. p. 13-18 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Linzen, T 2016, Issues in evaluating semantic spaces using word analogies. in Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 13-18, 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany, 8/7/16. https://doi.org/10.18653/v1/w16-2503

Linzen T. Issues in evaluating semantic spaces using word analogies. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. Association for Computational Linguistics (ACL). 2016. p. 13-18. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/w16-2503

Linzen, Tal. / Issues in evaluating semantic spaces using word analogies. Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016. Association for Computational Linguistics (ACL), 2016. pp. 13-18 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{092f46b065384ecba40a7ebf7406fc6d,

title = "Issues in evaluating semantic spaces using word analogies",

abstract = "The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: It is considered desirable for a space to represent semantic relations as consistent vector offsets. We show that the method's reliance on cosine similarity conflates offset consistency with largely irrelevant neighborhood structure, and propose simple baselines that should be used to improve the utility of the method in vector space evaluation.",

author = "Tal Linzen",

note = "Publisher Copyright: {\textcopyright} 2016 Proceedings of the Annual Meeting of the Association for Computational Linguistics. All Rights Reserved.; 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 ; Conference date: 07-08-2016",

year = "2016",

doi = "10.18653/v1/w16-2503",

language = "English (US)",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "13--18",

booktitle = "Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016",

}

TY - GEN

T1 - Issues in evaluating semantic spaces using word analogies

AU - Linzen, Tal

PY - 2016

Y1 - 2016

N2 - The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: It is considered desirable for a space to represent semantic relations as consistent vector offsets. We show that the method's reliance on cosine similarity conflates offset consistency with largely irrelevant neighborhood structure, and propose simple baselines that should be used to improve the utility of the method in vector space evaluation.

AB - The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: It is considered desirable for a space to represent semantic relations as consistent vector offsets. We show that the method's reliance on cosine similarity conflates offset consistency with largely irrelevant neighborhood structure, and propose simple baselines that should be used to improve the utility of the method in vector space evaluation.

UR - http://www.scopus.com/inward/record.url?scp=85073167942&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85073167942&partnerID=8YFLogxK

U2 - 10.18653/v1/w16-2503

DO - 10.18653/v1/w16-2503

M3 - Conference contribution

AN - SCOPUS:85073167942

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 13

EP - 18

BT - Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016

PB - Association for Computational Linguistics (ACL)

T2 - 1st Workshop on Evaluating Vector-Space Representations for NLP, RepEval 2016 at the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016

Y2 - 7 August 2016

ER -

Issues in evaluating semantic spaces using word analogies

Abstract

Publication series

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this