TY - GEN
T1 - Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation
AU - Yaman, Dogucan
AU - Eyiokur, Fevziye Irem
AU - Barmann, Leonard
AU - Akti, Seymanur
AU - Ekenel, Hazim Kemal
AU - Waibel, Alexander
N1 - Publisher Copyright:
© 2024 IEEE.
PY - 2024
Y1 - 2024
N2 - In the task of talking face generation, the objective is to generate a face video with lips synchronized to the corresponding audio while preserving visual details and identity information. Current methods face the challenge of learning accurate lip synchronization while avoiding detrimental effects on visual quality, as well as robustly evaluating such synchronization. To tackle these problems, we propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. Moreover, leveraging AV-HuBERT's features, we introduce three novel lip synchronization evaluation metrics, aiming to provide a comprehensive assessment of lip synchronization performance. Experimental results, along with a detailed ablation study, demonstrate the effectiveness of our approach and the utility of the proposed evaluation metrics.
AB - In the task of talking face generation, the objective is to generate a face video with lips synchronized to the corresponding audio while preserving visual details and identity information. Current methods face the challenge of learning accurate lip synchronization while avoiding detrimental effects on visual quality, as well as robustly evaluating such synchronization. To tackle these problems, we propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. Moreover, leveraging AV-HuBERT's features, we introduce three novel lip synchronization evaluation metrics, aiming to provide a comprehensive assessment of lip synchronization performance. Experimental results, along with a detailed ablation study, demonstrate the effectiveness of our approach and the utility of the proposed evaluation metrics.
UR - http://www.scopus.com/inward/record.url?scp=85206487112&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85206487112&partnerID=8YFLogxK
U2 - 10.1109/CVPRW63382.2024.00607
DO - 10.1109/CVPRW63382.2024.00607
M3 - Conference contribution
AN - SCOPUS:85206487112
T3 - IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
SP - 6003
EP - 6013
BT - Proceedings - 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2024
PB - IEEE Computer Society
T2 - 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2024
Y2 - 16 June 2024 through 22 June 2024
ER -