Many-shot Jailbreaking

Cem Anil; Esin Durmus; Nina Panickssery; Mrinank Sharma; Joe Benton; Sandipan Kundu; Joshua Batson; Meg Tong; Jesse Mu; Daniel Ford; Fracesco Mosconi; Rajashree Agrawal; Rylan Schaeffer; Naomi Bashkansky; Samuel Svenningsen; Mike Lambert; Ansh Radhakrishnan; Carson Denison; Evan J. Hubinger; Yuntao Bai; Trenton Bricken; Timothy Maxwell; Nicholas Schiefer; James Sully; Alex Tamkin; Tamera Lanhan; Karina Nguyen; Tomasz Korbak; Jared Kaplan; Deep Ganguli; Samuel R. Bowman; Ethan Perez; Roger Baker Grosse; David Duvenaud

Many-shot Jailbreaking

Cem Anil, Esin Durmus, Nina Panickssery, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel Ford, Fracesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J. Hubinger, Yuntao BaiTrenton Bricken, Timothy Maxwell, Nicholas Schiefer, James Sully, Alex Tamkin, Tamera Lanhan, Karina Nguyen, Tomasz Korbak, Jared Kaplan, Deep Ganguli, Samuel R. Bowman, Ethan Perez, Roger Baker Grosse, David Duvenaud

Research output: Contribution to journal › Conference article › peer-review

Abstract

We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This attack is newly feasible with the larger context windows recently deployed by language model providers like Google DeepMind, OpenAI and Anthropic. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.

Original language	English (US)
Journal	Advances in Neural Information Processing Systems
Volume	37
State	Published - 2024
Event	38th Conference on Neural Information Processing Systems, NeurIPS 2024 - Vancouver, Canada Duration: Dec 9 2024 → Dec 15 2024

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems
Signal Processing

Cite this

Anil, C, Durmus, E, Panickssery, N, Sharma, M, Benton, J, Kundu, S, Batson, J, Tong, M, Mu, J, Ford, D, Mosconi, F, Agrawal, R, Schaeffer, R, Bashkansky, N, Svenningsen, S, Lambert, M, Radhakrishnan, A, Denison, C, Hubinger, EJ, Bai, Y, Bricken, T, Maxwell, T, Schiefer, N, Sully, J, Tamkin, A, Lanhan, T, Nguyen, K, Korbak, T, Kaplan, J, Ganguli, D, Bowman, SR, Perez, E, Grosse, RB & Duvenaud, D 2024, 'Many-shot Jailbreaking', Advances in Neural Information Processing Systems, vol. 37.

@article{f0791dac79e142cfb60363b8f2e8b6d7,

title = "Many-shot Jailbreaking",

abstract = "We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This attack is newly feasible with the larger context windows recently deployed by language model providers like Google DeepMind, OpenAI and Anthropic. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.",

author = "Cem Anil and Esin Durmus and Nina Panickssery and Mrinank Sharma and Joe Benton and Sandipan Kundu and Joshua Batson and Meg Tong and Jesse Mu and Daniel Ford and Fracesco Mosconi and Rajashree Agrawal and Rylan Schaeffer and Naomi Bashkansky and Samuel Svenningsen and Mike Lambert and Ansh Radhakrishnan and Carson Denison and Hubinger, {Evan J.} and Yuntao Bai and Trenton Bricken and Timothy Maxwell and Nicholas Schiefer and James Sully and Alex Tamkin and Tamera Lanhan and Karina Nguyen and Tomasz Korbak and Jared Kaplan and Deep Ganguli and Bowman, {Samuel R.} and Ethan Perez and Grosse, {Roger Baker} and David Duvenaud",

note = "Publisher Copyright: {\textcopyright} 2024 Neural information processing systems foundation. All rights reserved.; 38th Conference on Neural Information Processing Systems, NeurIPS 2024 ; Conference date: 09-12-2024 Through 15-12-2024",

year = "2024",

language = "English (US)",

volume = "37",

journal = "Advances in Neural Information Processing Systems",

issn = "1049-5258",

publisher = "Neural information processing systems foundation",

}

TY - JOUR

T1 - Many-shot Jailbreaking

AU - Anil, Cem

AU - Durmus, Esin

AU - Panickssery, Nina

AU - Sharma, Mrinank

AU - Benton, Joe

AU - Kundu, Sandipan

AU - Batson, Joshua

AU - Tong, Meg

AU - Mu, Jesse

AU - Ford, Daniel

AU - Mosconi, Fracesco

AU - Agrawal, Rajashree

AU - Schaeffer, Rylan

AU - Bashkansky, Naomi

AU - Svenningsen, Samuel

AU - Lambert, Mike

AU - Radhakrishnan, Ansh

AU - Denison, Carson

AU - Hubinger, Evan J.

AU - Bai, Yuntao

AU - Bricken, Trenton

AU - Maxwell, Timothy

AU - Schiefer, Nicholas

AU - Sully, James

AU - Tamkin, Alex

AU - Lanhan, Tamera

AU - Nguyen, Karina

AU - Korbak, Tomasz

AU - Kaplan, Jared

AU - Ganguli, Deep

AU - Bowman, Samuel R.

AU - Perez, Ethan

AU - Grosse, Roger Baker

AU - Duvenaud, David

PY - 2024

Y1 - 2024

N2 - We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This attack is newly feasible with the larger context windows recently deployed by language model providers like Google DeepMind, OpenAI and Anthropic. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.

AB - We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This attack is newly feasible with the larger context windows recently deployed by language model providers like Google DeepMind, OpenAI and Anthropic. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.

UR - http://www.scopus.com/inward/record.url?scp=105000485865&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=105000485865&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:105000485865

SN - 1049-5258

VL - 37

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

T2 - 38th Conference on Neural Information Processing Systems, NeurIPS 2024

Y2 - 9 December 2024 through 15 December 2024

ER -

Many-shot Jailbreaking

Abstract

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this