Abstract
Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen in assembly, some due to long-read data. Related strains still were challenging for assembly and genome recovery through binning, as was assembly quality for the latter. Profilers markedly matured, with taxon profilers and binners excelling at higher bacterial ranks, but underperforming for viruses and Archaea. Clinical pathogen detection results revealed a need to improve reproducibility. Runtime and memory usage analyses identified efficient programs, including top performers with other metrics. The results identify challenges and guide researchers in selecting methods for analyses.
| Original language | English |
|---|---|
| Journal | Nature Methods |
| Volume | 19 |
| Issue number | 4 |
| Pages (from-to) | 429-440 |
| ISSN | 1548-7091 |
| DOIs | |
| Publication status | Published - 2022 |
Bibliographical note
Publisher Copyright:© 2022, The Author(s).
Cite this
- APA
- Standard
- Harvard
- Vancouver
- Author
- BIBTEX
- RIS
Critical Assessment of Metagenome Interpretation : the second round of challenges. / Meyer, Fernando; Fritz, Adrian; Deng, Zhi Luo; Koslicki, David; Lesker, Till Robin; Gurevich, Alexey; Robertson, Gary; Alser, Mohammed; Antipov, Dmitry; Beghini, Francesco; Bertrand, Denis; Brito, Jaqueline J.; Brown, C. Titus; Buchmann, Jan; Buluç, Aydin; Chen, Bo; Chikhi, Rayan; Clausen, Philip T.L.C.; Cristian, Alexandru; Dabrowski, Piotr Wojciech; Darling, Aaron E.; Egan, Rob; Eskin, Eleazar; Georganas, Evangelos; Goltsman, Eugene; Gray, Melissa A.; Hansen, Lars Hestbjerg; Hofmeyr, Steven; Huang, Pingqin; Irber, Luiz; Jia, Huijue; Jørgensen, Tue Sparholt; Kieser, Silas D.; Klemetsen, Terje; Kola, Axel; Kolmogorov, Mikhail; Korobeynikov, Anton; Kwan, Jason; LaPierre, Nathan; Lemaitre, Claire; Li, Chenhao; Limasset, Antoine; Malcher-Miranda, Fabio; Mangul, Serghei; Marcelino, Vanessa R.; Marchet, Camille; Marijon, Pierre; Meleshko, Dmitry; Mende, Daniel R.; Milanese, Alessio; Nagarajan, Niranjan; Nissen, Jakob; Nurk, Sergey; Oliker, Leonid; Paoli, Lucas; Peterlongo, Pierre; Piro, Vitor C.; Porter, Jacob S.; Rasmussen, Simon; Rees, Evan R.; Reinert, Knut; Renard, Bernhard; Robertsen, Espen Mikal; Rosen, Gail L.; Ruscheweyh, Hans Joachim; Sarwal, Varuni; Segata, Nicola; Seiler, Enrico; Shi, Lizhen; Sun, Fengzhu; Sunagawa, Shinichi; Sørensen, Søren Johannes; Thomas, Ashleigh; Tong, Chengxuan; Trajkovski, Mirko; Tremblay, Julien; Uritskiy, Gherman; Vicedomini, Riccardo; Wang, Zhengyang; Wang, Ziye; Wang, Zhong; Warren, Andrew; Willassen, Nils Peder; Yelick, Katherine; You, Ronghui; Zeller, Georg; Zhao, Zhengqiao; Zhu, Shanfeng; Zhu, Jie; Garrido-Oter, Ruben; Gastmeier, Petra; Hacquard, Stephane; Häußler, Susanne; Khaledi, Ariane; Maechler, Friederike; Mesny, Fantin; Radutoiu, Simona; Schulze-Lefert, Paul; Smit, Nathiana; Strowig, Till; Bremges, Andreas; Sczyrba, Alexander; McHardy, Alice Carolyn.
In: Nature Methods, Vol. 19, No. 4, 2022, p. 429-440.Research output: Contribution to journal › Journal article › Research › peer-review
}
TY - JOUR
T1 - Critical Assessment of Metagenome Interpretation
T2 - the second round of challenges
AU - Meyer, Fernando
AU - Fritz, Adrian
AU - Deng, Zhi Luo
AU - Koslicki, David
AU - Lesker, Till Robin
AU - Gurevich, Alexey
AU - Robertson, Gary
AU - Alser, Mohammed
AU - Antipov, Dmitry
AU - Beghini, Francesco
AU - Bertrand, Denis
AU - Brito, Jaqueline J.
AU - Brown, C. Titus
AU - Buchmann, Jan
AU - Buluç, Aydin
AU - Chen, Bo
AU - Chikhi, Rayan
AU - Clausen, Philip T.L.C.
AU - Cristian, Alexandru
AU - Dabrowski, Piotr Wojciech
AU - Darling, Aaron E.
AU - Egan, Rob
AU - Eskin, Eleazar
AU - Georganas, Evangelos
AU - Goltsman, Eugene
AU - Gray, Melissa A.
AU - Hansen, Lars Hestbjerg
AU - Hofmeyr, Steven
AU - Huang, Pingqin
AU - Irber, Luiz
AU - Jia, Huijue
AU - Jørgensen, Tue Sparholt
AU - Kieser, Silas D.
AU - Klemetsen, Terje
AU - Kola, Axel
AU - Kolmogorov, Mikhail
AU - Korobeynikov, Anton
AU - Kwan, Jason
AU - LaPierre, Nathan
AU - Lemaitre, Claire
AU - Li, Chenhao
AU - Limasset, Antoine
AU - Malcher-Miranda, Fabio
AU - Mangul, Serghei
AU - Marcelino, Vanessa R.
AU - Marchet, Camille
AU - Marijon, Pierre
AU - Meleshko, Dmitry
AU - Mende, Daniel R.
AU - Milanese, Alessio
AU - Nagarajan, Niranjan
AU - Nissen, Jakob
AU - Nurk, Sergey
AU - Oliker, Leonid
AU - Paoli, Lucas
AU - Peterlongo, Pierre
AU - Piro, Vitor C.
AU - Porter, Jacob S.
AU - Rasmussen, Simon
AU - Rees, Evan R.
AU - Reinert, Knut
AU - Renard, Bernhard
AU - Robertsen, Espen Mikal
AU - Rosen, Gail L.
AU - Ruscheweyh, Hans Joachim
AU - Sarwal, Varuni
AU - Segata, Nicola
AU - Seiler, Enrico
AU - Shi, Lizhen
AU - Sun, Fengzhu
AU - Sunagawa, Shinichi
AU - Sørensen, Søren Johannes
AU - Thomas, Ashleigh
AU - Tong, Chengxuan
AU - Trajkovski, Mirko
AU - Tremblay, Julien
AU - Uritskiy, Gherman
AU - Vicedomini, Riccardo
AU - Wang, Zhengyang
AU - Wang, Ziye
AU - Wang, Zhong
AU - Warren, Andrew
AU - Willassen, Nils Peder
AU - Yelick, Katherine
AU - You, Ronghui
AU - Zeller, Georg
AU - Zhao, Zhengqiao
AU - Zhu, Shanfeng
AU - Zhu, Jie
AU - Garrido-Oter, Ruben
AU - Gastmeier, Petra
AU - Hacquard, Stephane
AU - Häußler, Susanne
AU - Khaledi, Ariane
AU - Maechler, Friederike
AU - Mesny, Fantin
AU - Radutoiu, Simona
AU - Schulze-Lefert, Paul
AU - Smit, Nathiana
AU - Strowig, Till
AU - Bremges, Andreas
AU - Sczyrba, Alexander
AU - McHardy, Alice Carolyn
N1 - Publisher Copyright: © 2022, The Author(s).
PY - 2022
Y1 - 2022
N2 - Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen in assembly, some due to long-read data. Related strains still were challenging for assembly and genome recovery through binning, as was assembly quality for the latter. Profilers markedly matured, with taxon profilers and binners excelling at higher bacterial ranks, but underperforming for viruses and Archaea. Clinical pathogen detection results revealed a need to improve reproducibility. Runtime and memory usage analyses identified efficient programs, including top performers with other metrics. The results identify challenges and guide researchers in selecting methods for analyses.
AB - Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen in assembly, some due to long-read data. Related strains still were challenging for assembly and genome recovery through binning, as was assembly quality for the latter. Profilers markedly matured, with taxon profilers and binners excelling at higher bacterial ranks, but underperforming for viruses and Archaea. Clinical pathogen detection results revealed a need to improve reproducibility. Runtime and memory usage analyses identified efficient programs, including top performers with other metrics. The results identify challenges and guide researchers in selecting methods for analyses.
U2 - 10.1038/s41592-022-01431-4
DO - 10.1038/s41592-022-01431-4
M3 - Journal article
C2 - 35396482
AN - SCOPUS:85127703163
VL - 19
SP - 429
EP - 440
JO - Nature Methods
JF - Nature Methods
SN - 1548-7091
IS - 4
ER -