OdiEnCorp 2.0: Odia-English Parallel Corpus for Machine Translation

Shantipriya Parida, Satya Ranjan Dash, Ondřej Bojar, Petr Motlicek, Priyanka Pattnaik, Debasish Kumar Mallick


Abstract
The preparation of parallel corpora is a challenging task, particularly for languages that suffer from under-representation in the digital world. In a multi-lingual country like India, the need for such parallel corpora is stringent for several low-resource languages. In this work, we provide an extended English-Odia parallel corpus, OdiEnCorp 2.0, aiming particularly at Neural Machine Translation (NMT) systems which will help translate English↔Odia. OdiEnCorp 2.0 includes existing English-Odia corpora and we extended the collection by several other methods of data acquisition: parallel data scraping from many websites, including Odia Wikipedia, but also optical character recognition (OCR) to extract parallel data from scanned images. Our OCR-based data extraction approach for building a parallel corpus is suitable for other low resource languages that lack in online content. The resulting OdiEnCorp 2.0 contains 98,302 sentences and 1.69 million English and 1.47 million Odia tokens. To the best of our knowledge, OdiEnCorp 2.0 is the largest Odia-English parallel corpus covering different domains and available freely for non-commercial and research purposes.
Anthology ID:
2020.wildre-1.3
Volume:
Proceedings of the WILDRE5– 5th Workshop on Indian Language Data: Resources and Evaluation
Month:
May
Year:
2020
Address:
Marseille, France
Venues:
LREC | WILDRE | WS
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
14–19
Language:
English
URL:
https://www.aclweb.org/anthology/2020.wildre-1.3
DOI:
Bib Export formats:
BibTeX MODS XML EndNote
PDF:
http://aclanthology.lst.uni-saarland.de/2020.wildre-1.3.pdf