Position: Why We Must Rethink Empirical Research in Machine Learning

Moritz Herrmann; F. Julian D. Lange; Katharina Eggensperger; Giuseppe Casalicchio; Marcel Wever; Matthias Feurer; David Rügamer; Eyke Hüllermeier; Anne-Laure Boulesteix; Bernd Bischl

doi:10.48550/arXiv.2405.02200

Details

Original language	English
Title of host publication	Proceedings of the international conference on machine learning
Publication status	E-pub ahead of print - Jul 2024

Abstract

We warn against a common but incomplete understanding of empirical research in machine learning that leads to non-replicable results, makes findings unreliable, and threatens to undermine progress in the field. To overcome this alarming situation, we call for more awareness of the plurality of ways of gaining knowledge experimentally but also of some epistemic limitations. In particular, we argue most current empirical machine learning research is fashioned as confirmatory research while it should rather be considered exploratory.

Cite this

Position: Why We Must Rethink Empirical Research in Machine Learning. / Herrmann, Moritz; Lange, F. Julian D.; Eggensperger, Katharina et al.
Proceedings of the international conference on machine learning. 2024.

Research output: Chapter in book/report/conference proceeding › Contribution to book/anthology › Research › peer review

Herrmann, M, Lange, FJD, Eggensperger, K, Casalicchio, G, Wever, M, Feurer, M, Rügamer, D, Hüllermeier, E, Boulesteix, A-L & Bischl, B 2024, Position: Why We Must Rethink Empirical Research in Machine Learning. in Proceedings of the international conference on machine learning. https://doi.org/10.48550/arXiv.2405.02200

Herrmann, M., Lange, F. J. D., Eggensperger, K., Casalicchio, G., Wever, M., Feurer, M., Rügamer, D., Hüllermeier, E., Boulesteix, A.-L., & Bischl, B. (2024). Position: Why We Must Rethink Empirical Research in Machine Learning. In Proceedings of the international conference on machine learning Advance online publication. https://doi.org/10.48550/arXiv.2405.02200

Herrmann M, Lange FJD, Eggensperger K, Casalicchio G, Wever M, Feurer M et al. Position: Why We Must Rethink Empirical Research in Machine Learning. In Proceedings of the international conference on machine learning. 2024 Epub 2024 Jul. doi: 10.48550/arXiv.2405.02200

Herrmann, Moritz ; Lange, F. Julian D. ; Eggensperger, Katharina et al. / Position: Why We Must Rethink Empirical Research in Machine Learning. Proceedings of the international conference on machine learning. 2024.

Download

@inbook{ebdbce8eaa0d407e938b0688d929b989,

title = "Position: Why We Must Rethink Empirical Research in Machine Learning",

abstract = "We warn against a common but incomplete understanding of empirical research in machine learning that leads to non-replicable results, makes findings unreliable, and threatens to undermine progress in the field. To overcome this alarming situation, we call for more awareness of the plurality of ways of gaining knowledge experimentally but also of some epistemic limitations. In particular, we argue most current empirical machine learning research is fashioned as confirmatory research while it should rather be considered exploratory.",

author = "Moritz Herrmann and Lange, {F. Julian D.} and Katharina Eggensperger and Giuseppe Casalicchio and Marcel Wever and Matthias Feurer and David R{\"u}gamer and Eyke H{\"u}llermeier and Anne-Laure Boulesteix and Bernd Bischl",

note = "DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.",

year = "2024",

month = jul,

doi = "10.48550/arXiv.2405.02200",

language = "English",

booktitle = "Proceedings of the international conference on machine learning",

}

Download

TY - CHAP

T1 - Position: Why We Must Rethink Empirical Research in Machine Learning

AU - Herrmann, Moritz

AU - Lange, F. Julian D.

AU - Eggensperger, Katharina

AU - Casalicchio, Giuseppe

AU - Wever, Marcel

AU - Feurer, Matthias

AU - Rügamer, David

AU - Hüllermeier, Eyke

AU - Boulesteix, Anne-Laure

AU - Bischl, Bernd

N1 - DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.

PY - 2024/7

Y1 - 2024/7

N2 - We warn against a common but incomplete understanding of empirical research in machine learning that leads to non-replicable results, makes findings unreliable, and threatens to undermine progress in the field. To overcome this alarming situation, we call for more awareness of the plurality of ways of gaining knowledge experimentally but also of some epistemic limitations. In particular, we argue most current empirical machine learning research is fashioned as confirmatory research while it should rather be considered exploratory.

AB - We warn against a common but incomplete understanding of empirical research in machine learning that leads to non-replicable results, makes findings unreliable, and threatens to undermine progress in the field. To overcome this alarming situation, we call for more awareness of the plurality of ways of gaining knowledge experimentally but also of some epistemic limitations. In particular, we argue most current empirical machine learning research is fashioned as confirmatory research while it should rather be considered exploratory.

U2 - 10.48550/arXiv.2405.02200

DO - 10.48550/arXiv.2405.02200

M3 - Contribution to book/anthology

BT - Proceedings of the international conference on machine learning

ER -

Research@Leibniz University

Position: Why We Must Rethink Empirical Research in Machine Learning

Authors

Research Organisations

Details

Abstract

Cite this

By the same author(s)

Best Arm Identification with Retroactively Increased Sampling Budget for More Resource-Efficient HPO

On the Importance of Initialization in Active Learning

Annotation uncertainty in the context of grammatical change

Configuration and Evaluation

PyExperimenter: Easily distribute experiments and track results