Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics

Zhe Zhao, Tao Liu, Shen Li, Bofang Li, Xiaoyong Du


Abstract
The existing word representation methods mostly limit their information source to word co-occurrence statistics. In this paper, we introduce ngrams into four representation methods: SGNS, GloVe, PPMI matrix, and its SVD factorization. Comprehensive experiments are conducted on word analogy and similarity tasks. The results show that improved word representations are learned from ngram co-occurrence statistics. We also demonstrate that the trained ngram representations are useful in many aspects such as finding antonyms and collocations. Besides, a novel approach of building co-occurrence matrix is proposed to alleviate the hardware burdens brought by ngrams.
Anthology ID:
D17-1023
Erratum e1:
D17-1023e1
Volume:
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
244–253
Language:
URL:
https://www.aclweb.org/anthology/D17-1023
DOI:
10.18653/v1/D17-1023
Bib Export formats:
BibTeX MODS XML EndNote
PDF:
http://aclanthology.lst.uni-saarland.de/D17-1023.pdf