TY - GEN
T1 - OpenROAD-Assistant
T2 - 6th ACM/IEEE International Symposium on Machine Learning for CAD, MLCAD 2024
AU - Sharma, Utsav
AU - Wu, Bing Yue
AU - Kankipati, Sai Rahul Dhanvi
AU - Chhabria, Vidya A.
AU - Rovinski, Austin
N1 - Publisher Copyright:
© 2024 ACM.
PY - 2024/9/9
Y1 - 2024/9/9
N2 - Large language models (LLMs) have shown significant potential in serving as domain-specific chatbots. Recently, these models have emerged as powerful tools for chip design, providing both natural language responses and script generation for domain-specific inquiries. Previous work has demonstrated the effectiveness of LLMs in assisting with physical design automation; however, these approaches often rely on proprietary tools, APIs, technologies, and designs. As a result, access to these models is extremely limited, particularly for new chip designers who could greatly benefit from a design assistant. This paper introduces OpenROAD-Assistant, an open-source chatbot for OpenROAD that relies only on public data and responds to queries in either prose or Python script using the OpenROAD APIs. OpenROAD-Assistant leverages the Llama3-8B foundation model and employs retrieval-aware fine-tuning (RAFT) to respond to physical design-specific questions for OpenROAD. Notably, OpenROAD-Assistant outperforms other foundational models such as ChatGPT3.5, ChatGPT4, Code Llama, Claude3, and other ablation study baselines on the measured metrics (pass@k for scripting and BERTScore/BARTScore for question-answering). OpenROAD-Assistant achieves a 77% pass@1 score, 80% pass@3 score for scripting, and it achieves a 98% BERTScore and 96% BARTScore on question-answering.
AB - Large language models (LLMs) have shown significant potential in serving as domain-specific chatbots. Recently, these models have emerged as powerful tools for chip design, providing both natural language responses and script generation for domain-specific inquiries. Previous work has demonstrated the effectiveness of LLMs in assisting with physical design automation; however, these approaches often rely on proprietary tools, APIs, technologies, and designs. As a result, access to these models is extremely limited, particularly for new chip designers who could greatly benefit from a design assistant. This paper introduces OpenROAD-Assistant, an open-source chatbot for OpenROAD that relies only on public data and responds to queries in either prose or Python script using the OpenROAD APIs. OpenROAD-Assistant leverages the Llama3-8B foundation model and employs retrieval-aware fine-tuning (RAFT) to respond to physical design-specific questions for OpenROAD. Notably, OpenROAD-Assistant outperforms other foundational models such as ChatGPT3.5, ChatGPT4, Code Llama, Claude3, and other ablation study baselines on the measured metrics (pass@k for scripting and BERTScore/BARTScore for question-answering). OpenROAD-Assistant achieves a 77% pass@1 score, 80% pass@3 score for scripting, and it achieves a 98% BERTScore and 96% BARTScore on question-answering.
UR - http://www.scopus.com/inward/record.url?scp=85204966127&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85204966127&partnerID=8YFLogxK
U2 - 10.1145/3670474.3685960
DO - 10.1145/3670474.3685960
M3 - Conference contribution
AN - SCOPUS:85204966127
T3 - MLCAD 2024 - Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD
BT - MLCAD 2024 - Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD
PB - Association for Computing Machinery, Inc
Y2 - 9 September 2024 through 11 September 2024
ER -