Assessment of chemistry knowledge in large language models that generate code

Andrew D. White, Glen M. Hocky, Heta A. Gandhi, Mehrad Ansari, Sam Cox, Geemi P. Wellawatte, Subarna Sasmal, Ziyue Yang, Kangxin Liu, Yuvraj Singh, Willmor J. Peña Ccoa

Research output: Contribution to journalArticlepeer-review


In this work, we investigate the question: do code-generating large language models know chemistry? Our results indicate, mostly yes. To evaluate this, we introduce an expandable framework for evaluating chemistry knowledge in these models, through prompting models to solve chemistry problems posed as coding tasks. To do so, we produce a benchmark set of problems, and evaluate these models based on correctness of code by automated testing and evaluation by experts. We find that recent LLMs are able to write correct code across a variety of topics in chemistry and their accuracy can be increased by 30 percentage points via prompt engineering strategies, like putting copyright notices at the top of files. Our dataset and evaluation tools are open source which can be contributed to or built upon by future researchers, and will serve as a community resource for evaluating the performance of new models as they emerge. We also describe some good practices for employing LLMs in chemistry. The general success of these models demonstrates that their impact on chemistry teaching and research is poised to be enormous.

Original languageEnglish (US)
Pages (from-to)368-376
Number of pages9
JournalDigital Discovery
Issue number2
StatePublished - Apr 1 2023

ASJC Scopus subject areas

  • Chemistry (miscellaneous)


Dive into the research topics of 'Assessment of chemistry knowledge in large language models that generate code'. Together they form a unique fingerprint.

Cite this