TY - JOUR
T1 - Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization
AU - Ren, Mengwei
AU - Dey, Neel
AU - Fishbaugh, James
AU - Gerig, Guido
N1 - Funding Information:
Manuscript received December 31, 2020; accepted February 9, 2021. Date of publication February 16, 2021; date of current version June 1, 2021. This work was supported by the National Institutes of Health grants under Grant 1R01DA038215-01A1, Grant R01-HD055741-12, Grant 1R01HD088125-01A1, Grant 1R01MH118362-01, Grant 2R01EY013178-15, Grant R01EY030770-01A1, Grant R01ES032294, Grant R01MH122447, Grant 1R34DA050287, and Grant R01EB021391. (Mengwei Ren and Neel Dey contributed equally to this work.) (Corresponding author: Mengwei Ren.) The authors are with the Department of Computer Science and Engineering, New York University Tandon School of Engineering, Brooklyn, NY 11201 USA (e-mail: mengwei.ren@nyu.edu; neel.dey@nyu.edu; james.fishbaugh@nyu.edu; gerig@nyu.edu). Digital Object Identifier 10.1109/TMI.2021.3059726 Fig. 1. Unpaired samples across medical imaging scanners illustrating inter-device appearance variability.
Publisher Copyright:
© 1982-2012 IEEE.
PY - 2021/6
Y1 - 2021/6
N2 - Deep networks are now ubiquitous in large-scale multi-center imaging studies. However, the direct aggregation of images across sites is contraindicated for downstream statistical and deep learning-based image analysis due to inconsistent contrast, resolution, and noise. To this end, in the absence of paired data, variations of Cycle-consistent Generative Adversarial Networks have been used to harmonize image sets between a source and target domain. Importantly, these methods are prone to instability, contrast inversion, intractable manipulation of pathology, and steganographic mappings which limit their reliable adoption in real-world medical imaging. In this work, based on an underlying assumption that morphological shape is consistent across imaging sites, we propose a segmentation-renormalized image translation framework to reduce inter-scanner heterogeneity while preserving anatomical layout. We replace the affine transformations used in the normalization layers within generative networks with trainable scale and shift parameters conditioned on jointly learned anatomical segmentation embeddings to modulate features at every level of translation. We evaluate our methodologies against recent baselines across several imaging modalities (T1w MRI, FLAIR MRI, and OCT) on datasets with and without lesions. Segmentation-renormalization for translation GANs yields superior image harmonization as quantified by Inception distances, demonstrates improved downstream utility via post-hoc segmentation accuracy, and improved robustness to translation perturbation and self-adversarial attacks.
AB - Deep networks are now ubiquitous in large-scale multi-center imaging studies. However, the direct aggregation of images across sites is contraindicated for downstream statistical and deep learning-based image analysis due to inconsistent contrast, resolution, and noise. To this end, in the absence of paired data, variations of Cycle-consistent Generative Adversarial Networks have been used to harmonize image sets between a source and target domain. Importantly, these methods are prone to instability, contrast inversion, intractable manipulation of pathology, and steganographic mappings which limit their reliable adoption in real-world medical imaging. In this work, based on an underlying assumption that morphological shape is consistent across imaging sites, we propose a segmentation-renormalized image translation framework to reduce inter-scanner heterogeneity while preserving anatomical layout. We replace the affine transformations used in the normalization layers within generative networks with trainable scale and shift parameters conditioned on jointly learned anatomical segmentation embeddings to modulate features at every level of translation. We evaluate our methodologies against recent baselines across several imaging modalities (T1w MRI, FLAIR MRI, and OCT) on datasets with and without lesions. Segmentation-renormalization for translation GANs yields superior image harmonization as quantified by Inception distances, demonstrates improved downstream utility via post-hoc segmentation accuracy, and improved robustness to translation perturbation and self-adversarial attacks.
KW - Unpaired image translation
KW - conditional normalization
KW - generative adversarial networks
KW - image harmonization
KW - image segmentation
UR - http://www.scopus.com/inward/record.url?scp=85100923769&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85100923769&partnerID=8YFLogxK
U2 - 10.1109/TMI.2021.3059726
DO - 10.1109/TMI.2021.3059726
M3 - Article
C2 - 33591913
AN - SCOPUS:85100923769
SN - 0278-0062
VL - 40
SP - 1519
EP - 1530
JO - IEEE Transactions on Medical Imaging
JF - IEEE Transactions on Medical Imaging
IS - 6
M1 - 9354797
ER -