Characterization of tumour heterogeneity through segmentation-free representation learning on multiplexed imaging data

High-dimensional multiplexed imaging can reveal the spatial organization of tumour tissues at the molecular level. However, owing to the scale and information complexity of the imaging data, it is challenging to discover and thoroughly characterize the heterogeneity of tumour microenvironments. Here we show that self-supervised representation learning on data from imaging mass cytometry can be leveraged to distinguish morphological differences in tumour microenvironments and to precisely characterize distinct microenvironment signatures. We used self-supervised masked image modelling to train a vision transformer that directly takes high-dimensional multiplexed mass-cytometry images. In contrast with traditional spatial analyses relying on cellular segmentation, the vision transformer is segmentation-free, uses pixel-level information, and retains information on the local morphology and biomarker distribution. By applying the vision transformer to a lung-tumour dataset, we identified and validated a monocytic signature that is associated with poor prognosis.