Stop Word Lists in Free Open-source Software Packages

Joel Nothman, Hanmin Qin, Roman Yurchak


Abstract
Open-source software packages for language processing often include stop word lists. Users may apply them without awareness of their surprising omissions (e.g. “hasn’t” but not “hadn’t”) and inclusions (“computer”), or their incompatibility with a particular tokenizer. Motivated by issues raised about the Scikit-learn stop list, we investigate variation among and consistency within 52 popular English-language stop lists, and propose strategies for mitigating these issues.
Anthology ID:
W18-2502
Volume:
Proceedings of Workshop for NLP Open Source Software (NLP-OSS)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Venues:
ACL | NLPOSS | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7–12
Language:
URL:
https://www.aclweb.org/anthology/W18-2502
DOI:
10.18653/v1/W18-2502
Bib Export formats:
BibTeX MODS XML EndNote
PDF:
http://aclanthology.lst.uni-saarland.de/W18-2502.pdf
Software:
 W18-2502.Software.zip