Orthographic and morphological processing for English-Arabic statistical machine translation

Ahmed El Kholy, Nizar Habash

Research output: Contribution to journalArticle

Abstract

Much of the work on statistical machine translation (SMT) from morphologically rich languages has shown that morphological tokenization and orthographic normalization help improve SMT quality because of the sparsity reduction they contribute. In this article, we study the effect of these processes on SMT when translating into a morphologically rich language, namely Arabic. We explore a space of tokenization schemes and normalization options. We also examine a set of six detokenization techniques and evaluate on detokenized and orthographically correct (enriched) output. Our results show that the best performing tokenization scheme is that of the Penn Arabic Treebank. Additionally, training on orthographically normalized (reduced) text then jointly enriching and detokenizing the output outperforms training on enriched text.

Original languageEnglish (US)
Pages (from-to)25-45
Number of pages21
JournalMachine Translation
Volume26
Issue number1-2
DOIs
StatePublished - Mar 1 2012

Keywords

  • Arabic language
  • Detokenization
  • Morphology
  • Statistical machine translation
  • Tokenization

ASJC Scopus subject areas

  • Software
  • Language and Linguistics
  • Linguistics and Language
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Orthographic and morphological processing for English-Arabic statistical machine translation'. Together they form a unique fingerprint.

  • Cite this