Measurement of inter-rater agreement for transient events using Monte Carlo sampled permutations

Robert G. Norman, Marc A. Scott

Research output: Contribution to journalArticlepeer-review


In this paper we demonstrate the adverse effect of serially observed data sequences containing transient events on the calculation of Cohen's κ as an index of inter-rater agreement in the detection of these events. We develop and use a Monte-Carlo-based permutation technique to produce an empiric distribution of κ in the presence of serial dependence. We find that the empiric confidence intervals for κ tend to be wider than parametrically derived intervals and in the case of longer event lengths, are markedly so. We evaluate the effect of number and length of events, and further, describe and evaluate three permutation methods which match specific rating situations. Finally, we apply these techniques to the measurement of inter-rater agreement for sleep disordered breathing events, a transient event identified during nocturnal polysomnography, for which traditionally computed confidence intervals for κ are incorrect.

Original languageEnglish (US)
Pages (from-to)931-942
Number of pages12
JournalStatistics in Medicine
Issue number4
StatePublished - Feb 20 2007


  • Inter-rater agreement
  • Monte Carlo
  • Permutation test
  • Transient events

ASJC Scopus subject areas

  • Epidemiology
  • Statistics and Probability


Dive into the research topics of 'Measurement of inter-rater agreement for transient events using Monte Carlo sampled permutations'. Together they form a unique fingerprint.

Cite this