Exploiting Dialect Identification in Automatic Dialectal Text Normalization

Bashar Alhafni; Sarah Al-Towaity; Ziyad Fawzy; Fatema Nassar; Fadhl Eryani; Houda Bouamor; Nizar Habash

Exploiting Dialect Identification in Automatic Dialectal Text Normalization

Bashar Alhafni, Sarah Al-Towaity, Ziyad Fawzy, Fatema Nassar, Fadhl Eryani, Houda Bouamor, Nizar Habash

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Dialectal Arabic is the primary spoken language used by native Arabic speakers in daily communication. The rise of social media platforms has notably expanded its use as a written language. However, Arabic dialects do not have standard orthographies. This, combined with the inherent noise in user-generated content on social media, presents a major challenge to NLP applications dealing with Dialectal Arabic. In this paper, we explore and report on the task of CODAfication, which aims to normalize Dialectal Arabic into the Conventional Orthography for Dialectal Arabic (CODA). We work with a unique parallel corpus of multiple Arabic dialects focusing on five major city dialects. We benchmark newly developed pretrained sequence-to-sequence models on the task of CODAfication. We further show that using dialect identification information improves the performance across all dialects. We make our code, data, and pretrained models publicly available.

Original language	English (US)
Title of host publication	ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference
Editors	Nizar Habash, Houda Bouamor, Ramy Eskander, Nadi Tomeh, Ibrahim Abu Farha, Ahmed Abdelali, Samia Touileb, Injy Hamed, Yaser Onaizan, Bashar Alhafni, Wissam Antoun, Salam Khalifa, Hatem Haddad, Imed Zitouni, Badr AlKhamissi, Rawan Almatham, Khalil Mrini
Publisher	Association for Computational Linguistics (ACL)
Pages	42-54
Number of pages	13
ISBN (Electronic)	9798891761322
State	Published - 2024
Event	2nd Arabic Natural Language Processing Conference, ArabicNLP 2024 - Bangkok, Thailand Duration: Aug 16 2024 → …

Publication series

Name	ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference

Conference

Conference	2nd Arabic Natural Language Processing Conference, ArabicNLP 2024
Country/Territory	Thailand
City	Bangkok
Period	8/16/24 → …

ASJC Scopus subject areas

Language and Linguistics
Computational Theory and Mathematics
Software
Linguistics and Language

Cite this

Alhafni, B., Al-Towaity, S., Fawzy, Z., Nassar, F., Eryani, F., Bouamor, H., & Habash, N. (2024). Exploiting Dialect Identification in Automatic Dialectal Text Normalization. In N. Habash, H. Bouamor, R. Eskander, N. Tomeh, I. A. Farha, A. Abdelali, S. Touileb, I. Hamed, Y. Onaizan, B. Alhafni, W. Antoun, S. Khalifa, H. Haddad, I. Zitouni, B. AlKhamissi, R. Almatham, & K. Mrini (Eds.), ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference (pp. 42-54). (ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference). Association for Computational Linguistics (ACL).

Exploiting Dialect Identification in Automatic Dialectal Text Normalization. / Alhafni, Bashar; Al-Towaity, Sarah; Fawzy, Ziyad et al.
ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference. ed. / Nizar Habash; Houda Bouamor; Ramy Eskander; Nadi Tomeh; Ibrahim Abu Farha; Ahmed Abdelali; Samia Touileb; Injy Hamed; Yaser Onaizan; Bashar Alhafni; Wissam Antoun; Salam Khalifa; Hatem Haddad; Imed Zitouni; Badr AlKhamissi; Rawan Almatham; Khalil Mrini. Association for Computational Linguistics (ACL), 2024. p. 42-54 (ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Alhafni, B, Al-Towaity, S, Fawzy, Z, Nassar, F, Eryani, F, Bouamor, H & Habash, N 2024, Exploiting Dialect Identification in Automatic Dialectal Text Normalization. in N Habash, H Bouamor, R Eskander, N Tomeh, IA Farha, A Abdelali, S Touileb, I Hamed, Y Onaizan, B Alhafni, W Antoun, S Khalifa, H Haddad, I Zitouni, B AlKhamissi, R Almatham & K Mrini (eds), ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference. ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 42-54, 2nd Arabic Natural Language Processing Conference, ArabicNLP 2024, Bangkok, Thailand, 8/16/24.

Alhafni B, Al-Towaity S, Fawzy Z, Nassar F, Eryani F, Bouamor H et al. Exploiting Dialect Identification in Automatic Dialectal Text Normalization. In Habash N, Bouamor H, Eskander R, Tomeh N, Farha IA, Abdelali A, Touileb S, Hamed I, Onaizan Y, Alhafni B, Antoun W, Khalifa S, Haddad H, Zitouni I, AlKhamissi B, Almatham R, Mrini K, editors, ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2024. p. 42-54. (ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference).

Alhafni, Bashar ; Al-Towaity, Sarah ; Fawzy, Ziyad et al. / Exploiting Dialect Identification in Automatic Dialectal Text Normalization. ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference. editor / Nizar Habash ; Houda Bouamor ; Ramy Eskander ; Nadi Tomeh ; Ibrahim Abu Farha ; Ahmed Abdelali ; Samia Touileb ; Injy Hamed ; Yaser Onaizan ; Bashar Alhafni ; Wissam Antoun ; Salam Khalifa ; Hatem Haddad ; Imed Zitouni ; Badr AlKhamissi ; Rawan Almatham ; Khalil Mrini. Association for Computational Linguistics (ACL), 2024. pp. 42-54 (ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference).

@inproceedings{84fd1a0377ad480da65d872385295e56,

title = "Exploiting Dialect Identification in Automatic Dialectal Text Normalization",

abstract = "Dialectal Arabic is the primary spoken language used by native Arabic speakers in daily communication. The rise of social media platforms has notably expanded its use as a written language. However, Arabic dialects do not have standard orthographies. This, combined with the inherent noise in user-generated content on social media, presents a major challenge to NLP applications dealing with Dialectal Arabic. In this paper, we explore and report on the task of CODAfication, which aims to normalize Dialectal Arabic into the Conventional Orthography for Dialectal Arabic (CODA). We work with a unique parallel corpus of multiple Arabic dialects focusing on five major city dialects. We benchmark newly developed pretrained sequence-to-sequence models on the task of CODAfication. We further show that using dialect identification information improves the performance across all dialects. We make our code, data, and pretrained models publicly available.",

author = "Bashar Alhafni and Sarah Al-Towaity and Ziyad Fawzy and Fatema Nassar and Fadhl Eryani and Houda Bouamor and Nizar Habash",

note = "Publisher Copyright: {\textcopyright}2024 Association for Computational Linguistics.; 2nd Arabic Natural Language Processing Conference, ArabicNLP 2024 ; Conference date: 16-08-2024",

year = "2024",

language = "English (US)",

series = "ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "42--54",

editor = "Nizar Habash and Houda Bouamor and Ramy Eskander and Nadi Tomeh and Farha, {Ibrahim Abu} and Ahmed Abdelali and Samia Touileb and Injy Hamed and Yaser Onaizan and Bashar Alhafni and Wissam Antoun and Salam Khalifa and Hatem Haddad and Imed Zitouni and Badr AlKhamissi and Rawan Almatham and Khalil Mrini",

booktitle = "ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference",

}

TY - GEN

T1 - Exploiting Dialect Identification in Automatic Dialectal Text Normalization

AU - Alhafni, Bashar

AU - Al-Towaity, Sarah

AU - Fawzy, Ziyad

AU - Nassar, Fatema

AU - Eryani, Fadhl

AU - Bouamor, Houda

AU - Habash, Nizar

PY - 2024

Y1 - 2024

N2 - Dialectal Arabic is the primary spoken language used by native Arabic speakers in daily communication. The rise of social media platforms has notably expanded its use as a written language. However, Arabic dialects do not have standard orthographies. This, combined with the inherent noise in user-generated content on social media, presents a major challenge to NLP applications dealing with Dialectal Arabic. In this paper, we explore and report on the task of CODAfication, which aims to normalize Dialectal Arabic into the Conventional Orthography for Dialectal Arabic (CODA). We work with a unique parallel corpus of multiple Arabic dialects focusing on five major city dialects. We benchmark newly developed pretrained sequence-to-sequence models on the task of CODAfication. We further show that using dialect identification information improves the performance across all dialects. We make our code, data, and pretrained models publicly available.

AB - Dialectal Arabic is the primary spoken language used by native Arabic speakers in daily communication. The rise of social media platforms has notably expanded its use as a written language. However, Arabic dialects do not have standard orthographies. This, combined with the inherent noise in user-generated content on social media, presents a major challenge to NLP applications dealing with Dialectal Arabic. In this paper, we explore and report on the task of CODAfication, which aims to normalize Dialectal Arabic into the Conventional Orthography for Dialectal Arabic (CODA). We work with a unique parallel corpus of multiple Arabic dialects focusing on five major city dialects. We benchmark newly developed pretrained sequence-to-sequence models on the task of CODAfication. We further show that using dialect identification information improves the performance across all dialects. We make our code, data, and pretrained models publicly available.

UR - http://www.scopus.com/inward/record.url?scp=85204306991&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85204306991&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85204306991

T3 - ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference

SP - 42

EP - 54

BT - ArabicNLP 2024 - 2nd Arabic Natural Language Processing Conference, Proceedings of the Conference

A2 - Habash, Nizar

A2 - Bouamor, Houda

A2 - Eskander, Ramy

A2 - Tomeh, Nadi

A2 - Farha, Ibrahim Abu

A2 - Abdelali, Ahmed

A2 - Touileb, Samia

A2 - Hamed, Injy

A2 - Onaizan, Yaser

A2 - Alhafni, Bashar

A2 - Antoun, Wissam

A2 - Khalifa, Salam

A2 - Haddad, Hatem

A2 - Zitouni, Imed

A2 - AlKhamissi, Badr

A2 - Almatham, Rawan

A2 - Mrini, Khalil

PB - Association for Computational Linguistics (ACL)

T2 - 2nd Arabic Natural Language Processing Conference, ArabicNLP 2024

Y2 - 16 August 2024

ER -

Exploiting Dialect Identification in Automatic Dialectal Text Normalization

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this