Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization

Mengwei Ren, Neel Dey, James Fishbaugh, Guido Gerig

    Research output: Contribution to journalArticlepeer-review

    Abstract

    Deep networks are now ubiquitous in large-scale multi-center imaging studies. However, the direct aggregation of images across sites is contraindicated for downstream statistical and deep learning-based image analysis due to inconsistent contrast, resolution, and noise. To this end, in the absence of paired data, variations of Cycle-consistent Generative Adversarial Networks have been used to harmonize image sets between a source and target domain. Importantly, these methods are prone to instability, contrast inversion, intractable manipulation of pathology, and steganographic mappings which limit their reliable adoption in real-world medical imaging. In this work, based on an underlying assumption that morphological shape is consistent across imaging sites, we propose a segmentation-renormalized image translation framework to reduce inter-scanner heterogeneity while preserving anatomical layout. We replace the affine transformations used in the normalization layers within generative networks with trainable scale and shift parameters conditioned on jointly learned anatomical segmentation embeddings to modulate features at every level of translation. We evaluate our methodologies against recent baselines across several imaging modalities (T1w MRI, FLAIR MRI, and OCT) on datasets with and without lesions. Segmentation-renormalization for translation GANs yields superior image harmonization as quantified by Inception distances, demonstrates improved downstream utility via post-hoc segmentation accuracy, and improved robustness to translation perturbation and self-adversarial attacks.

    Original languageEnglish (US)
    Article number9354797
    Pages (from-to)1519-1530
    Number of pages12
    JournalIEEE Transactions on Medical Imaging
    Volume40
    Issue number6
    DOIs
    StatePublished - Jun 2021

    Keywords

    • Unpaired image translation
    • conditional normalization
    • generative adversarial networks
    • image harmonization
    • image segmentation

    ASJC Scopus subject areas

    • Software
    • Radiological and Ultrasound Technology
    • Computer Science Applications
    • Electrical and Electronic Engineering

    Fingerprint

    Dive into the research topics of 'Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization'. Together they form a unique fingerprint.

    Cite this