Multi-Modal Models for Concrete and Abstract Concept Meaning

Felix Hill, Roi Reichart, Anna Korhonen


Abstract
Multi-modal models that learn semantic representations from both linguistic and perceptual input outperform language-only models on a range of evaluations, and better reflect human concept acquisition. Most perceptual input to such models corresponds to concrete noun concepts and the superiority of the multi-modal approach has only been established when evaluating on such concepts. We therefore investigate which concepts can be effectively learned by multi-modal models. We show that concreteness determines both which linguistic features are most informative and the impact of perceptual input in such models. We then introduce ridge regression as a means of propagating perceptual information from concrete nouns to more abstract concepts that is more robust than previous approaches. Finally, we present weighted gram matrix combination, a means of combining representations from distinct modalities that outperforms alternatives when both modalities are sufficiently rich.
Anthology ID:
Q14-1023
Volume:
Transactions of the Association for Computational Linguistics, Volume 2
Month:
Year:
2014
Address:
Venue:
TACL
SIG:
Publisher:
Note:
Pages:
285–296
Language:
URL:
https://www.aclweb.org/anthology/Q14-1023
DOI:
10.1162/tacl_a_00183
Bib Export formats:
BibTeX MODS XML EndNote
PDF:
http://aclanthology.lst.uni-saarland.de/Q14-1023.pdf