An Empirical Study of Generation Order for Machine Translation

William Chan, Mitchell Stern, Jamie Kiros, Jakob Uszkoreit


Abstract
In this work, we present an empirical study of generation order for machine translation. Building on recent advances in insertion-based modeling, we first introduce a soft order-reward framework that enables us to train models to follow arbitrary oracle generation policies. We then make use of this framework to explore a large variety of generation orders, including uninformed orders, location-based orders, frequency-based orders, content-based orders, and model-based orders. Curiously, we find that for the WMT’14 English German and WMT’18 English Chinese translation tasks, order does not have a substantial impact on output quality. Moreover, for English German, we even discover that unintuitive orderings such as alphabetical and shortest-first can match the performance of a standard Transformer, suggesting that traditional left-to-right generation may not be necessary to achieve high performance.
Anthology ID:
2020.emnlp-main.464
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5764–5773
Language:
URL:
https://www.aclweb.org/anthology/2020.emnlp-main.464
DOI:
10.18653/v1/2020.emnlp-main.464
Bib Export formats:
BibTeX MODS XML EndNote
PDF:
http://aclanthology.lst.uni-saarland.de/2020.emnlp-main.464.pdf
Optional supplementary material:
 2020.emnlp-main.464.OptionalSupplementaryMaterial.pdf