Understanding the Detrimental Class-level Effects of Data Augmentation

Polina Kirichenko, Mark Ibrahim, Randall Balestriero, Diane Bouchacourt, Ramakrishna Vedantam, Hamed Firooz, Andrew Gordon Wilson

Research output: Contribution to journalConference articlepeer-review


Data augmentation (DA) encodes invariance and provides implicit regularization critical to a model's performance in image classification tasks.However, while DA improves average accuracy, recent studies have shown that its impact can be highly class dependent: achieving optimal average accuracy comes at the cost of significantly hurting individual class accuracy by as much as 20% on ImageNet.There has been little progress in resolving class-level accuracy drops due to a limited understanding of these effects.In this work, we present a framework for understanding how DA interacts with class-level learning dynamics.Using higher-quality multi-label annotations on ImageNet, we systematically categorize the affected classes and find that the majority are inherently ambiguous, co-occur, or involve fine-grained distinctions, while DA controls the model's bias towards one of the closely related classes.While many of the previously reported performance drops are explained by multi-label annotations, our analysis of class confusions reveals other sources of accuracy degradation.We show that simple class-conditional augmentation strategies informed by our framework improve performance on the negatively affected classes.

Original languageEnglish (US)
JournalAdvances in Neural Information Processing Systems
StatePublished - 2023
Event37th Conference on Neural Information Processing Systems, NeurIPS 2023 - New Orleans, United States
Duration: Dec 10 2023Dec 16 2023

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing


Dive into the research topics of 'Understanding the Detrimental Class-level Effects of Data Augmentation'. Together they form a unique fingerprint.

Cite this