Overview of the SustaiNLP 2020 Shared Task

Alex Wang, Thomas Wolf


Abstract
We describe the SustaiNLP 2020 shared task: efficient inference on the SuperGLUE benchmark (Wang et al., 2019). Participants are evaluated based on performance on the benchmark as well as energy consumed in making predictions on the test sets. We describe the task, its organization, and the submitted systems. Across the six submissions to the shared task, participants achieved efficiency gains of 20× over a standard BERT (Devlin et al., 2019) baseline, while losing less than an absolute point in performance.
Anthology ID:
2020.sustainlp-1.24
Volume:
Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing
Month:
November
Year:
2020
Address:
Online
Venues:
EMNLP | sustainlp
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
174–178
Language:
URL:
https://www.aclweb.org/anthology/2020.sustainlp-1.24
DOI:
10.18653/v1/2020.sustainlp-1.24
Bib Export formats:
BibTeX MODS XML EndNote
PDF:
http://aclanthology.lst.uni-saarland.de/2020.sustainlp-1.24.pdf