A Study on Robustness to Perturbations for Representations of Environmental Sound

Sangeeta Srivastava, Ho Hsiang Wu, Joao Rulff, Magdalena Fuentes, Mark Cartwright, Claudio Silva, Anish Arora, Juan Pablo Bello

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Audio applications involving environmental sound analysis increasingly use general-purpose audio representations, also known as embeddings, for transfer learning. Recently, Holistic Evaluation of Audio Representations (HEAR) evaluated twenty-nine embedding models on nineteen diverse tasks. However, the evaluation's effectiveness depends on the variation already captured within a given dataset. Therefore, for a given data domain, it is unclear how the representations would be affected by the variations caused by myriad microphones' range and acoustic conditions - commonly known as channel effects. We aim to extend HEAR to evaluate invariance to channel effects in this work. To accomplish this, we imitate channel effects by injecting perturbations to the audio signal and measure the shift in the new (perturbed) embeddings with three distance measures, making the evaluation domain-dependent but not task-dependent. Combined with the downstream performance, it helps us make a more informed prediction of how robust the embeddings are to the channel effects. We evaluate two embeddings - YAMNet, and OpenL3 on monophonic (UrbanSound8K) and polyphonic (SONYC-UST) urban datasets. We show that one distance measure does not suffice in such task-independent evaluation. Although Fréchet Audio Distance (FAD) correlates with the trend of the performance drop in the downstream task most accurately, we show that we need to study FAD in conjunction with the other distances to get a clear understanding of the overall effect of the perturbation. In terms of the embedding performance, we find OpenL3 to be more robust than YAMNet, which aligns with the HEAR evaluation.

Original languageEnglish (US)
Title of host publication30th European Signal Processing Conference, EUSIPCO 2022 - Proceedings
PublisherEuropean Signal Processing Conference, EUSIPCO
Pages125-129
Number of pages5
ISBN (Electronic)9789082797091
StatePublished - 2022
Event30th European Signal Processing Conference, EUSIPCO 2022 - Belgrade, Serbia
Duration: Aug 29 2022Sep 2 2022

Publication series

NameEuropean Signal Processing Conference
Volume2022-August
ISSN (Print)2219-5491

Conference

Conference30th European Signal Processing Conference, EUSIPCO 2022
Country/TerritorySerbia
CityBelgrade
Period8/29/229/2/22

Keywords

  • acoustic perturbations
  • robust audio embeddings
  • Self-supervised learning
  • transfer learning
  • urban sound

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'A Study on Robustness to Perturbations for Representations of Environmental Sound'. Together they form a unique fingerprint.

Cite this