Quality and Accountability of ChatGPT in Health Care in Low- and Middle-Income Countries: Simulated Patient Study

Yafei Si; Yuyi Yang; Xi Wang; Jiaqi Zu; Xi Chen; Xiaojing Fan; Ruopeng An; Sen Gong

doi:10.2196/56121

Quality and Accountability of ChatGPT in Health Care in Low- and Middle-Income Countries: Simulated Patient Study

Yafei Si, Yuyi Yang, Xi Wang, Jiaqi Zu, Xi Chen, Xiaojing Fan, Ruopeng An, Sen Gong

Social Work

Research output: Contribution to journal › Article › peer-review

Abstract

Using simulated patients to mimic 9 established noncommunicable and infectious diseases, we assessed ChatGPT’s performance in treatment recommendations for common diseases in low- and middle-income countries. ChatGPT had a high level of accuracy in both correct diagnoses (20/27, 74%) and medication prescriptions (22/27, 82%) but a concerning level of unnecessary or harmful medications (23/27, 85%) even with correct diagnoses. ChatGPT performed better in managing noncommunicable diseases than infectious ones. These results highlight the need for cautious AI integration in health care systems to ensure quality and safety.

Original language	English (US)
Article number	e56121
Journal	Journal of medical Internet research
Volume	26
DOIs	https://doi.org/10.2196/56121
State	Published - 2024

Keywords

AI
AI integration
ChatGPT
LMIC
artificial intelligence
effectiveness
generative AI
health care
low- and middle-income countries
medication prescription
noncommunicable diseases
patient study
prescription
quality
quality and safety
reliability
simulated patient

ASJC Scopus subject areas

Health Informatics

Access to Document

10.2196/56121

Cite this

@article{fcc390c5be3340d891968db04fda627b,

title = "Quality and Accountability of ChatGPT in Health Care in Low- and Middle-Income Countries: Simulated Patient Study",

abstract = "Using simulated patients to mimic 9 established noncommunicable and infectious diseases, we assessed ChatGPT{\textquoteright}s performance in treatment recommendations for common diseases in low- and middle-income countries. ChatGPT had a high level of accuracy in both correct diagnoses (20/27, 74%) and medication prescriptions (22/27, 82%) but a concerning level of unnecessary or harmful medications (23/27, 85%) even with correct diagnoses. ChatGPT performed better in managing noncommunicable diseases than infectious ones. These results highlight the need for cautious AI integration in health care systems to ensure quality and safety.",

keywords = "AI, AI integration, ChatGPT, LMIC, artificial intelligence, effectiveness, generative AI, health care, low- and middle-income countries, medication prescription, noncommunicable diseases, patient study, prescription, quality, quality and safety, reliability, simulated patient",

author = "Yafei Si and Yuyi Yang and Xi Wang and Jiaqi Zu and Xi Chen and Xiaojing Fan and Ruopeng An and Sen Gong",

note = "Publisher Copyright: {\textcopyright}Yafei Si, Yuyi Yang, Xi Wang, Jiaqi Zu, Xi Chen, Xiaojing Fan, Ruopeng An, Sen Gong.",

year = "2024",

doi = "10.2196/56121",

language = "English (US)",

volume = "26",

journal = "Journal of medical Internet research",

issn = "1439-4456",

publisher = "JMIR Publications Inc.",

}

TY - JOUR

T1 - Quality and Accountability of ChatGPT in Health Care in Low- and Middle-Income Countries

T2 - Simulated Patient Study

AU - Si, Yafei

AU - Yang, Yuyi

AU - Wang, Xi

AU - Zu, Jiaqi

AU - Chen, Xi

AU - Fan, Xiaojing

AU - An, Ruopeng

AU - Gong, Sen

N1 - Publisher Copyright: ©Yafei Si, Yuyi Yang, Xi Wang, Jiaqi Zu, Xi Chen, Xiaojing Fan, Ruopeng An, Sen Gong.

PY - 2024

Y1 - 2024

N2 - Using simulated patients to mimic 9 established noncommunicable and infectious diseases, we assessed ChatGPT’s performance in treatment recommendations for common diseases in low- and middle-income countries. ChatGPT had a high level of accuracy in both correct diagnoses (20/27, 74%) and medication prescriptions (22/27, 82%) but a concerning level of unnecessary or harmful medications (23/27, 85%) even with correct diagnoses. ChatGPT performed better in managing noncommunicable diseases than infectious ones. These results highlight the need for cautious AI integration in health care systems to ensure quality and safety.

AB - Using simulated patients to mimic 9 established noncommunicable and infectious diseases, we assessed ChatGPT’s performance in treatment recommendations for common diseases in low- and middle-income countries. ChatGPT had a high level of accuracy in both correct diagnoses (20/27, 74%) and medication prescriptions (22/27, 82%) but a concerning level of unnecessary or harmful medications (23/27, 85%) even with correct diagnoses. ChatGPT performed better in managing noncommunicable diseases than infectious ones. These results highlight the need for cautious AI integration in health care systems to ensure quality and safety.

KW - AI

KW - AI integration

KW - ChatGPT

KW - LMIC

KW - artificial intelligence

KW - effectiveness

KW - generative AI

KW - health care

KW - low- and middle-income countries

KW - medication prescription

KW - noncommunicable diseases

KW - patient study

KW - prescription

KW - quality

KW - quality and safety

KW - reliability

KW - simulated patient

UR - http://www.scopus.com/inward/record.url?scp=85203519526&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85203519526&partnerID=8YFLogxK

U2 - 10.2196/56121

DO - 10.2196/56121

M3 - Article

C2 - 39250188

AN - SCOPUS:85203519526

SN - 1439-4456

VL - 26

JO - Journal of medical Internet research

JF - Journal of medical Internet research

M1 - e56121

ER -

Quality and Accountability of ChatGPT in Health Care in Low- and Middle-Income Countries: Simulated Patient Study

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this