SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge

Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, Manfred Pinkal


Abstract
This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98%.
Anthology ID:
S18-1119
Volume:
Proceedings of The 12th International Workshop on Semantic Evaluation
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Venue:
*SEMEVAL
SIGs:
SIGLEX | SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
747–757
Language:
URL:
https://www.aclweb.org/anthology/S18-1119
DOI:
10.18653/v1/S18-1119
Bib Export formats:
BibTeX MODS XML EndNote
PDF:
http://aclanthology.lst.uni-saarland.de/S18-1119.pdf