Benchmarking Large Language Models for Automated Verilog RTL Code Generation

Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Hammond Pearce, Benjamin Tan, Ramesh Karri, Brendan Dolan-Gavitt, Siddharth Garg

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Automating hardware design could obviate a signif-icant amount of human error from the engineering process and lead to fewer errors. Verilog is a popular hardware description language to model and design digital systems, thus generating Verilog code is a critical first step. Emerging large language models (LLMs) are able to write high-quality code in other programming languages. In this paper, we characterize the ability of LLMs to generate useful Verilog. For this, we fine-tune pre-trained LLMs on Verilog datasets collected from GitHub and Verilog textbooks. We construct an evaluation framework comprising test-benches for functional analysis and a flow to test the syntax of Verilog code generated in response to problems of varying difficulty. Our findings show that across our problem scenarios, the fine-tuning results in LLMs more capable of producing syntactically correct code (25.9% overall). Further, when analyzing functional correctness, a fine-tuned open-source CodeGen LLM can outperform the state-of-the-art commercial Codex LLM (6.5% overall). We release our training/evaluation scripts and LLM checkpoints as open source contributions.

Original languageEnglish (US)
Title of host publication2023 Design, Automation and Test in Europe Conference and Exhibition, DATE 2023 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9783981926378
DOIs
StatePublished - 2023
Event2023 Design, Automation and Test in Europe Conference and Exhibition, DATE 2023 - Antwerp, Belgium
Duration: Apr 17 2023Apr 19 2023

Publication series

NameProceedings -Design, Automation and Test in Europe, DATE
Volume2023-April
ISSN (Print)1530-1591

Conference

Conference2023 Design, Automation and Test in Europe Conference and Exhibition, DATE 2023
Country/TerritoryBelgium
CityAntwerp
Period4/17/234/19/23

Keywords

  • GPT
  • LLM
  • Transformers
  • Verilog

ASJC Scopus subject areas

  • General Engineering

Fingerprint

Dive into the research topics of 'Benchmarking Large Language Models for Automated Verilog RTL Code Generation'. Together they form a unique fingerprint.

Cite this