LAST LAYER RE-TRAINING IS SUFFICIENT FOR ROBUSTNESS TO SPURIOUS CORRELATIONS

Polina Kirichenko; Pavel Izmailov; Andrew Gordon Wilson

LAST LAYER RE-TRAINING IS SUFFICIENT FOR ROBUSTNESS TO SPURIOUS CORRELATIONS

Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson

Computer Science

Research output: Contribution to conference › Paper › peer-review

Abstract

Neural network classifiers can largely rely on simple spurious features, such as backgrounds, to make predictions. However, even in these cases, we show that they still often learn core features associated with the desired attributes of the data, contrary to recent findings. Inspired by this insight, we demonstrate that simple last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks, but with profoundly lower complexity and computational expenses. Moreover, we show that last layer retraining on large ImageNet-trained models can also significantly reduce reliance on background and texture information, improving robustness to covariate shift, after only minutes of training on a single GPU.

Original language	English (US)
State	Published - 2023
Event	11th International Conference on Learning Representations, ICLR 2023 - Kigali, Rwanda Duration: May 1 2023 → May 5 2023

Conference

Conference	11th International Conference on Learning Representations, ICLR 2023
Country/Territory	Rwanda
City	Kigali
Period	5/1/23 → 5/5/23

ASJC Scopus subject areas

Language and Linguistics
Computer Science Applications
Education
Linguistics and Language

Cite this

@conference{f09655c1c0874b68b7cb901ab7b491f6,

title = "LAST LAYER RE-TRAINING IS SUFFICIENT FOR ROBUSTNESS TO SPURIOUS CORRELATIONS",

abstract = "Neural network classifiers can largely rely on simple spurious features, such as backgrounds, to make predictions. However, even in these cases, we show that they still often learn core features associated with the desired attributes of the data, contrary to recent findings. Inspired by this insight, we demonstrate that simple last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks, but with profoundly lower complexity and computational expenses. Moreover, we show that last layer retraining on large ImageNet-trained models can also significantly reduce reliance on background and texture information, improving robustness to covariate shift, after only minutes of training on a single GPU.",

author = "Polina Kirichenko and Pavel Izmailov and Wilson, {Andrew Gordon}",

note = "Publisher Copyright: {\textcopyright} 2023 11th International Conference on Learning Representations, ICLR 2023. All rights reserved.; 11th International Conference on Learning Representations, ICLR 2023 ; Conference date: 01-05-2023 Through 05-05-2023",

year = "2023",

language = "English (US)",

}

TY - CONF

T1 - LAST LAYER RE-TRAINING IS SUFFICIENT FOR ROBUSTNESS TO SPURIOUS CORRELATIONS

AU - Kirichenko, Polina

AU - Izmailov, Pavel

AU - Wilson, Andrew Gordon

PY - 2023

Y1 - 2023

N2 - Neural network classifiers can largely rely on simple spurious features, such as backgrounds, to make predictions. However, even in these cases, we show that they still often learn core features associated with the desired attributes of the data, contrary to recent findings. Inspired by this insight, we demonstrate that simple last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks, but with profoundly lower complexity and computational expenses. Moreover, we show that last layer retraining on large ImageNet-trained models can also significantly reduce reliance on background and texture information, improving robustness to covariate shift, after only minutes of training on a single GPU.

AB - Neural network classifiers can largely rely on simple spurious features, such as backgrounds, to make predictions. However, even in these cases, we show that they still often learn core features associated with the desired attributes of the data, contrary to recent findings. Inspired by this insight, we demonstrate that simple last layer retraining can match or outperform state-of-the-art approaches on spurious correlation benchmarks, but with profoundly lower complexity and computational expenses. Moreover, we show that last layer retraining on large ImageNet-trained models can also significantly reduce reliance on background and texture information, improving robustness to covariate shift, after only minutes of training on a single GPU.

UR - http://www.scopus.com/inward/record.url?scp=85199907232&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85199907232&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85199907232

T2 - 11th International Conference on Learning Representations, ICLR 2023

Y2 - 1 May 2023 through 5 May 2023

ER -

LAST LAYER RE-TRAINING IS SUFFICIENT FOR ROBUSTNESS TO SPURIOUS CORRELATIONS

Abstract

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this