Ho Hsiang Wu, Magdalena Fuentes, Juan P. Bello

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Music information is often conveyed or recorded across multiple data modalities including but not limited to audio, images, text and scores. However, music information retrieval research has almost exclusively focused on single modality recognition, requiring development of separate models for each modality. Some multi-modal works require multiple coexisting modalities given to the model as inputs, constraining the use of these models to the few cases where data from all modalities are available. To the best of our knowledge, no existing model has the ability to take inputs from varying modalities, e.g. images or sounds, and classify them into unified music categories. We explore the use of cross-modal retrieval as a pretext task to learn modality-agnostic representations, which can then be used as inputs to classifiers that are independent of modality. We select instrument classification as an example task for our study as both visual and audio components provide relevant semantic information. We train music instrument classifiers that can take both images or sounds as input, and perform comparably to sound-only or image-only classifiers. Furthermore, we explore the case when there is limited labeled data for a given modality, and the impact in performance by using labeled data from other modalities. We are able to achieve almost 70% of best performing system in a zero-shot setting. We provide a detailed analysis of experimental results to understand the potential and limitations of the approach, and discuss future steps towards modality-agnostic classifiers.

Original languageEnglish (US)
Title of host publicationSMC 2021 - Proceedings of the 18th Sound and Music Computing Conference
EditorsDavide Andrea Mauro, Simone Spagnol, Andrea Valle
PublisherSound and music Computing network
Number of pages8
ISBN (Electronic)9788894541540
StatePublished - 2021
Event18th Sound and Music Computing Conference, SMC 2021 - Virtual, Online
Duration: Jun 29 2021Jul 1 2021

Publication series

NameProceedings of the Sound and Music Computing Conferences
ISSN (Electronic)2518-3672


Conference18th Sound and Music Computing Conference, SMC 2021
CityVirtual, Online

ASJC Scopus subject areas

  • Music
  • Computer Science Applications
  • Media Technology


Dive into the research topics of 'EXPLORING MODALITY-AGNOSTIC REPRESENTATIONS for MUSIC CLASSIFICATION'. Together they form a unique fingerprint.

Cite this