Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, Rob Fergus

Research output: Contribution to journalArticlepeer-review


In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised learning has led to major advances in representation learning and statistical generation. In the life sciences, the anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity. The resulting model contains information about biological properties in its representations. The representations are learned from sequence data alone. The learned representation space has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and can be identified by linear projections. Representation learning produces features that generalize across a range of applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and improving state-of-the-art features for long-range contact prediction.

Original languageEnglish (US)
Article numbere2016239118
JournalProceedings of the National Academy of Sciences of the United States of America
Issue number15
StatePublished - Apr 13 2021


  • Deep learning
  • Generative biology
  • Protein language model
  • Representation learning
  • Synthetic biology

ASJC Scopus subject areas

  • General


Dive into the research topics of 'Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences'. Together they form a unique fingerprint.

Cite this