Neyman-Pearson classification algorithms and NP receiver operating characteristics

Xin Tong, Yang Feng, Jingyi Jessica Li

Research output: Contribution to journalArticlepeer-review


In many binary classification applications, such as disease diagnosis and spam detection, practitioners commonly face the need to limit type I error (that is, the conditional probability of misclassifying a class 0 observation as class 1) so that it remains below a desired threshold. To address this need, the Neyman-Pearson (NP) classification paradigm is a natural choice; it minimizes type II error (that is, the conditional probability of misclassifying a class 1 observation as class 0)while enforcing an upper bound, a, on the type I error.Despite its century-long history in hypothesis testing, the NP paradigm has not been well recognized and implemented in classification schemes. Common practices that directly limit the empirical type I error to no more than a do not satisfy the type I error control objective because the resulting classifiers are likely to have type I errors much larger than a, and the NP paradigm has not been properly implemented in practice. We develop the first umbrella algorithm that implements the NP paradigm for all scoringtype classification methods, such as logistic regression, support vector machines, and random forests. Powered by this algorithm, we propose a novel graphical tool for NP classification methods: NP receiver operating characteristic (NP-ROC) bands motivated by the popular ROC curves. NP-ROC bands will help choose a in a data-adaptive way and compare different NP classifiers. We demonstrate the use and properties of the NP umbrella algorithm and NP-ROC bands, available in the R package nproc, through simulation and real data studies.

Original languageEnglish (US)
Article numbereaao1659
JournalScience Advances
Issue number2
StatePublished - Feb 2018

ASJC Scopus subject areas

  • General


Dive into the research topics of 'Neyman-Pearson classification algorithms and NP receiver operating characteristics'. Together they form a unique fingerprint.

Cite this