Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach

Md Faijul Amin, Sergey M. Plis, Eswar Damaraju, Devon Hjelm, Kyunghyun Cho, Vince D. Calhoun

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this work, we study a novel approach of deep neural machine translation to find linkage between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). The idea is to consider two different imaging views of the same brain like two different languages conveying some common concepts or facts. An important aspect of the translation model is an attention network module that learns alignment between features from fMRI and sMRI. We use independent component analysis (ICA) based features for the translation model. Our study shows significant group differences between healthy controls and patients with schizophrenia in the learned alignments. Furthermore, this novel approach reveals a group differential relation between a cognitive score (attention and vigilance) and alignments that could not be found when individual modality of data were considered.

Original languageEnglish (US)
Title of host publication2016 IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1-4
Number of pages4
ISBN (Electronic)9781467399197
DOIs
StatePublished - Apr 25 2016
EventIEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Santa Fe, United States
Duration: Mar 6 2016Mar 8 2016

Publication series

NameProceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation
Volume2016-April

Other

OtherIEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016
CountryUnited States
CitySanta Fe
Period3/6/163/8/16

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Fingerprint Dive into the research topics of 'Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach'. Together they form a unique fingerprint.

  • Cite this

    Amin, M. F., Plis, S. M., Damaraju, E., Hjelm, D., Cho, K., & Calhoun, V. D. (2016). Multimodal fusion of brain structural and functional imaging with a deep neural machine translation approach. In 2016 IEEE Southwest Symposium on Image Analysis and Interpretation, SSIAI 2016 - Proceedings (pp. 1-4). [7459160] (Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation; Vol. 2016-April). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SSIAI.2016.7459160