repro_eval: A Python Interface to Reproducibility Measures of System-Oriented IR Experiments

Timo Breuer; Nicola Ferro; Maria Maistro; Philipp Schaer

doi:10.1007/978-3-030-72240-1_51

repro_eval: A Python Interface to Reproducibility Measures of System-Oriented IR Experiments

Timo Breuer^*, Nicola Ferro, Maria Maistro, Philipp Schaer

^*Corresponding author for this work

Machine Learning

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

13 Citations (Scopus)

15 Downloads (Pure)

Abstract

In this work we introduce repro_eval - a tool for reactive reproducibility studies of system-oriented Information Retrieval (IR) experiments. The corresponding Python package provides IR researchers with measures for different levels of reproduction when evaluating their systems’ outputs. By offering an easily extensible interface, we hope to stimulate common practices when conducting a reproducibility study of system-oriented IR experiments.

Original language	English
Title of host publication	Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings
Editors	Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, Fabrizio Sebastiani
Number of pages	6
Publisher	Springer
Publication date	2021
Pages	481-486
ISBN (Print)	9783030722395
DOIs	https://doi.org/10.1007/978-3-030-72240-1_51
Publication status	Published - 2021
Event	43rd European Conference on Information Retrieval, ECIR 2021 - Virtual, Online Duration: 28 Mar 2021 → 1 Apr 2021

Conference

Conference	43rd European Conference on Information Retrieval, ECIR 2021
City	Virtual, Online
Period	28/03/2021 → 01/04/2021

Series	Lecture Notes in Computer Science
Volume	12657 LNCS
ISSN	0302-9743

Bibliographical note

Publisher Copyright:
© 2021, Springer Nature Switzerland AG.

Keywords

Evaluation
Replicability
Reproducibility

Access to Document

10.1007/978-3-030-72240-1_51

FulltextSubmitted manuscript, 426 KB

Cite this

Breuer, T., Ferro, N., Maistro, M., & Schaer, P. (2021). repro_eval: A Python Interface to Reproducibility Measures of System-Oriented IR Experiments. In D. Hiemstra, M-F. Moens, J. Mothe, R. Perego, M. Potthast, & F. Sebastiani (Eds.), Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings (pp. 481-486). Springer. Lecture Notes in Computer Science Vol. 12657 LNCS https://doi.org/10.1007/978-3-030-72240-1_51

repro_eval : A Python Interface to Reproducibility Measures of System-Oriented IR Experiments. / Breuer, Timo; Ferro, Nicola; Maistro, Maria; Schaer, Philipp.

Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings. ed. / Djoerd Hiemstra; Marie-Francine Moens; Josiane Mothe; Raffaele Perego; Martin Potthast; Fabrizio Sebastiani. Springer, 2021. p. 481-486 (Lecture Notes in Computer Science, Vol. 12657 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Breuer, T, Ferro, N, Maistro, M & Schaer, P 2021, repro_eval: A Python Interface to Reproducibility Measures of System-Oriented IR Experiments. in D Hiemstra, M-F Moens, J Mothe, R Perego, M Potthast & F Sebastiani (eds), Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings. Springer, Lecture Notes in Computer Science, vol. 12657 LNCS, pp. 481-486, 43rd European Conference on Information Retrieval, ECIR 2021, Virtual, Online, 28/03/2021. https://doi.org/10.1007/978-3-030-72240-1_51

Breuer T, Ferro N, Maistro M, Schaer P. repro_eval: A Python Interface to Reproducibility Measures of System-Oriented IR Experiments. In Hiemstra D, Moens M-F, Mothe J, Perego R, Potthast M, Sebastiani F, editors, Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings. Springer. 2021. p. 481-486. (Lecture Notes in Computer Science, Vol. 12657 LNCS). https://doi.org/10.1007/978-3-030-72240-1_51

Breuer, Timo ; Ferro, Nicola ; Maistro, Maria ; Schaer, Philipp. / repro_eval : A Python Interface to Reproducibility Measures of System-Oriented IR Experiments. Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings. editor / Djoerd Hiemstra ; Marie-Francine Moens ; Josiane Mothe ; Raffaele Perego ; Martin Potthast ; Fabrizio Sebastiani. Springer, 2021. pp. 481-486 (Lecture Notes in Computer Science, Vol. 12657 LNCS).

@inproceedings{281935e510424ba5993475f951dd0129,

title = "repro_eval: A Python Interface to Reproducibility Measures of System-Oriented IR Experiments",

abstract = "In this work we introduce repro_eval - a tool for reactive reproducibility studies of system-oriented Information Retrieval (IR) experiments. The corresponding Python package provides IR researchers with measures for different levels of reproduction when evaluating their systems{\textquoteright} outputs. By offering an easily extensible interface, we hope to stimulate common practices when conducting a reproducibility study of system-oriented IR experiments.",

keywords = "Evaluation, Replicability, Reproducibility",

author = "Timo Breuer and Nicola Ferro and Maria Maistro and Philipp Schaer",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; 43rd European Conference on Information Retrieval, ECIR 2021 ; Conference date: 28-03-2021 Through 01-04-2021",

year = "2021",

doi = "10.1007/978-3-030-72240-1_51",

language = "English",

isbn = "9783030722395",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "481--486",

editor = "Djoerd Hiemstra and Marie-Francine Moens and Josiane Mothe and Raffaele Perego and Martin Potthast and Fabrizio Sebastiani",

booktitle = "Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings",

address = "Switzerland",

}

TY - GEN

T1 - repro_eval

T2 - 43rd European Conference on Information Retrieval, ECIR 2021

AU - Breuer, Timo

AU - Ferro, Nicola

AU - Maistro, Maria

AU - Schaer, Philipp

PY - 2021

Y1 - 2021

N2 - In this work we introduce repro_eval - a tool for reactive reproducibility studies of system-oriented Information Retrieval (IR) experiments. The corresponding Python package provides IR researchers with measures for different levels of reproduction when evaluating their systems’ outputs. By offering an easily extensible interface, we hope to stimulate common practices when conducting a reproducibility study of system-oriented IR experiments.

AB - In this work we introduce repro_eval - a tool for reactive reproducibility studies of system-oriented Information Retrieval (IR) experiments. The corresponding Python package provides IR researchers with measures for different levels of reproduction when evaluating their systems’ outputs. By offering an easily extensible interface, we hope to stimulate common practices when conducting a reproducibility study of system-oriented IR experiments.

KW - Evaluation

KW - Replicability

KW - Reproducibility

U2 - 10.1007/978-3-030-72240-1_51

DO - 10.1007/978-3-030-72240-1_51

M3 - Article in proceedings

AN - SCOPUS:85107327580

SN - 9783030722395

T3 - Lecture Notes in Computer Science

SP - 481

EP - 486

BT - Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings

A2 - Hiemstra, Djoerd

A2 - Moens, Marie-Francine

A2 - Mothe, Josiane

A2 - Perego, Raffaele

A2 - Potthast, Martin

A2 - Sebastiani, Fabrizio

PB - Springer

Y2 - 28 March 2021 through 1 April 2021

ER -