TY - JOUR
T1 - Comprehensive benchmarking and ensemble approaches for metagenomic classifiers
AU - McIntyre, Alexa B.R.
AU - Ounit, Rachid
AU - Afshinnekoo, Ebrahim
AU - Prill, Robert J.
AU - Hénaff, Elizabeth
AU - Alexander, Noah
AU - Minot, Samuel S.
AU - Danko, David
AU - Foox, Jonathan
AU - Ahsanuddin, Sofia
AU - Tighe, Scott
AU - Hasan, Nur A.
AU - Subramanian, Poorani
AU - Moffat, Kelly
AU - Levy, Shawn
AU - Lonardi, Stefano
AU - Greenfield, Nick
AU - Colwell, Rita R.
AU - Rosen, Gail L.
AU - Mason, Christopher E.
N1 - Publisher Copyright:
© 2017 The Author(s).
PY - 2017/9/21
Y1 - 2017/9/21
N2 - Background: One of the main challenges in metagenomics is the identification of microorganisms in clinical and environmental samples. While an extensive and heterogeneous set of computational tools is available to classify microorganisms using whole-genome shotgun sequencing data, comprehensive comparisons of these methods are limited. Results: In this study, we use the largest-to-date set of laboratory-generated and simulated controls across 846 species to evaluate the performance of 11 metagenomic classifiers. Tools were characterized on the basis of their ability to identify taxa at the genus, species, and strain levels, quantify relative abundances of taxa, and classify individual reads to the species level. Strikingly, the number of species identified by the 11 tools can differ by over three orders of magnitude on the same datasets. Various strategies can ameliorate taxonomic misclassification, including abundance filtering, ensemble approaches, and tool intersection. Nevertheless, these strategies were often insufficient to completely eliminate false positives from environmental samples, which are especially important where they concern medically relevant species. Overall, pairing tools with different classification strategies (k-mer, alignment, marker) can combine their respective advantages. Conclusions: This study provides positive and negative controls, titrated standards, and a guide for selecting tools for metagenomic analyses by comparing ranges of precision, accuracy, and recall. We show that proper experimental design and analysis parameters can reduce false positives, provide greater resolution of species in complex metagenomic samples, and improve the interpretation of results.
AB - Background: One of the main challenges in metagenomics is the identification of microorganisms in clinical and environmental samples. While an extensive and heterogeneous set of computational tools is available to classify microorganisms using whole-genome shotgun sequencing data, comprehensive comparisons of these methods are limited. Results: In this study, we use the largest-to-date set of laboratory-generated and simulated controls across 846 species to evaluate the performance of 11 metagenomic classifiers. Tools were characterized on the basis of their ability to identify taxa at the genus, species, and strain levels, quantify relative abundances of taxa, and classify individual reads to the species level. Strikingly, the number of species identified by the 11 tools can differ by over three orders of magnitude on the same datasets. Various strategies can ameliorate taxonomic misclassification, including abundance filtering, ensemble approaches, and tool intersection. Nevertheless, these strategies were often insufficient to completely eliminate false positives from environmental samples, which are especially important where they concern medically relevant species. Overall, pairing tools with different classification strategies (k-mer, alignment, marker) can combine their respective advantages. Conclusions: This study provides positive and negative controls, titrated standards, and a guide for selecting tools for metagenomic analyses by comparing ranges of precision, accuracy, and recall. We show that proper experimental design and analysis parameters can reduce false positives, provide greater resolution of species in complex metagenomic samples, and improve the interpretation of results.
KW - Classification
KW - Comparison
KW - Ensemble methods
KW - Meta-classification
KW - Metagenomics
KW - Pathogen detection
KW - Shotgun sequencing
KW - Taxonomy
UR - http://www.scopus.com/inward/record.url?scp=85029756134&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85029756134&partnerID=8YFLogxK
U2 - 10.1186/s13059-017-1299-7
DO - 10.1186/s13059-017-1299-7
M3 - Article
C2 - 28934964
AN - SCOPUS:85029756134
SN - 1474-7596
VL - 18
JO - Genome biology
JF - Genome biology
IS - 1
M1 - 182
ER -