Alternating diffusion maps for multimodal data fusion

Ori Katz, Ronen Talmon, Yu Lun Lo, Hau Tieng Wu

Research output: Contribution to journalArticlepeer-review


The problem of information fusion from multiple data-sets acquired by multimodal sensors has drawn significant research attention over the years. In this paper, we focus on a particular problem setting consisting of a physical phenomenon or a system of interest observed by multiple sensors. We assume that all sensors measure some aspects of the system of interest with additional sensor-specific and irrelevant components. Our goal is to recover the variables relevant to the observed system and to filter out the nuisance effects of the sensor-specific variables. We propose an approach based on manifold learning, which is particularly suitable for problems with multiple modalities, since it aims to capture the intrinsic structure of the data and relies on minimal prior model knowledge. Specifically, we propose a nonlinear filtering scheme, which extracts the hidden sources of variability captured by two or more sensors, that are independent of the sensor-specific components. In addition to presenting a theoretical analysis, we demonstrate our technique on real measured data for the purpose of sleep stage assessment based on multiple, multimodal sensor measurements. We show that without prior knowledge on the different modalities and on the measured system, our method gives rise to a data-driven representation that is well correlated with the underlying sleep process and is robust to noise and sensor-specific effects.

Original languageEnglish (US)
Pages (from-to)346-360
Number of pages15
JournalInformation Fusion
StatePublished - Jan 2019


  • Alternating diffusion maps
  • Common manifold learning
  • Diffusion maps
  • Multimodal sensor fusion
  • Nonlinear-filtering

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Information Systems
  • Hardware and Architecture


Dive into the research topics of 'Alternating diffusion maps for multimodal data fusion'. Together they form a unique fingerprint.

Cite this