Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

Muhammad F. Walji, Elsbeth Kalenderian, Mark Piotrowski, Duong Tran, Krishna K. Kookal, Oluwabunmi Tokede, Joel M. White, Ram Vaderhobli, Rachel Ramoni, Paul C. Stark, Nicole S. Kimmes, Maxim Lagerweij, Vimla L. Patel

Research output: Contribution to journalArticlepeer-review


Objective: To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. Materials and methods: Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N= 32), semi-structured interviews (N= 36), and surveys (N= 35). Results: The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods. Discussion: Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. Conclusion: We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

Original languageEnglish (US)
Pages (from-to)361-367
Number of pages7
JournalInternational Journal of Medical Informatics
Issue number5
StatePublished - May 2014


  • EHR
  • Human factors
  • Methodology
  • Usability

ASJC Scopus subject areas

  • Health Informatics


Dive into the research topics of 'Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR'. Together they form a unique fingerprint.

Cite this