Can Wikipedia Categories Improve Masked Language Model Pretraining?

Diksha Meghwal, Katharina Kann, Iacer Calixto, Stanislaw Jastrzebski


Abstract
Pretrained language models have obtained impressive results for a large set of natural language understanding tasks. However, training these models is computationally expensive and requires huge amounts of data. Thus, it would be desirable to automatically detect groups of more or less important examples. Here, we investigate if we can leverage sources of information which are commonly overlooked, Wikipedia categories as listed in DBPedia, to identify useful or harmful data points during pretraining. We define an experimental setup in which we analyze correlations between language model perplexity on specific clusters and downstream NLP task performances during pretraining. Our experiments show that Wikipedia categories are not a good indicator of the importance of specific sentences for pretraining.
Anthology ID:
2020.winlp-1.19
Volume:
Proceedings of the The Fourth Widening Natural Language Processing Workshop
Month:
July
Year:
2020
Address:
Seattle, USA
Venues:
ACL | WS | WiNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
78
Language:
URL:
https://www.aclweb.org/anthology/2020.winlp-1.19
DOI:
10.18653/v1/2020.winlp-1.19
Bib Export formats:
BibTeX MODS XML EndNote
Video:
 http://slideslive.com/38929556