Visual TTR - Modelling Visual Question Answering in Type Theory with Records

Ronja Utescher


Abstract
In this paper, I will describe a system that was developed for the task of Visual Question Answering. The system uses the rich type universe of Type Theory with Records (TTR) to model both the utterances about the image, the image itself and classifications made related to the two. At its most basic, the decision of whether any given predicate can be assigned to an object in the image is delegated to a CNN. Consequently, images can be judged as evidence for propositions. The end result is a model whose application of perceptual classifiers to a given image is guided by the accompanying utterance.
Anthology ID:
W19-0602
Volume:
Proceedings of the 13th International Conference on Computational Semantics - Student Papers
Month:
May
Year:
2019
Address:
Gothenburg, Sweden
Venues:
IWCS | WS
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
9–14
Language:
URL:
https://www.aclweb.org/anthology/W19-0602
DOI:
10.18653/v1/W19-0602
Bib Export formats:
BibTeX MODS XML EndNote
PDF:
http://aclanthology.lst.uni-saarland.de/W19-0602.pdf