Cross-Task Feedback Fusion GAN for Joint MR-CT Synthesis and Segmentation of Target and Organs-at-Risk

Yiwen Zhang, Liming Zhong, Hai Shu, Zhenhui Dai, Kaiyi Zheng, Zefeiyun Chen, Qianjin Feng, Xuetao Wang, Wei Yang

Research output: Contribution to journalArticlepeer-review

Abstract

The synthesis of computed tomography (CT) images from magnetic resonance imaging (MR) images and segmentation of target and organs-at-risk (OARs) are two important tasks in MR-only radiotherapy treatment planning (RTP). Some methods have been proposed to utilize the paired MR and CT images for MR-CT synthesis or target and OARs segmentation. However, these methods usually handle synthesis and segmentation as two separate tasks, and ignore the inevitable registration errors in paired images after standard registration. In this article, we propose a cross-task feedback fusion generative adversarial network (CTFF-GAN) for joint MR-CT synthesis and segmentation of target and OARs to enhance each task's performance. Specifically, we propose a cross-task feedback fusion (CTFF) module to feedback the semantic information from the segmentation task to the synthesis task for the anatomical structure correction in synthetic CT images. Besides, we use CT images synthesized from MR images for multimodal segmentation to eliminate the registration errors. Moreover, we develop a multitask discriminator to urge the generator to devote more attention to the organ boundaries. Experiments on our nasopharyngeal carcinoma dataset show that CTFF-GAN achieves impressive performance with MAE of 70.69 ± 10.50 HU, SSIM of 0.755 ± 0.03, and PSNR of 27.44 ± 1.20 dB in synthetic CT, and the mean dice of 0.783 ± 0.075 in target and OARs segmentation. Our CTFF-GAN outperforms state-of-the-art methods in both the synthesis and segmentation tasks.

Original languageEnglish (US)
Pages (from-to)1246-1257
Number of pages12
JournalIEEE Transactions on Artificial Intelligence
Volume4
Issue number5
DOIs
StatePublished - Oct 1 2023

Keywords

  • Feedback fusion mechanism
  • MR-only radiotherapy treatment planning (RTP)
  • generative adversarial network
  • joint synthesis and segmentation

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'Cross-Task Feedback Fusion GAN for Joint MR-CT Synthesis and Segmentation of Target and Organs-at-Risk'. Together they form a unique fingerprint.

Cite this