Details
| Original language | English |
|---|---|
| Article number | 9382913 |
| Pages (from-to) | 3079-3090 |
| Number of pages | 12 |
| Journal | IEEE Transactions on Pattern Analysis and Machine Intelligence |
| Volume | 43 |
| Issue number | 9 |
| Early online date | 22 Mar 2021 |
| Publication status | Published - 1 Sept 2021 |
Abstract
Keywords
- cs.LG, cs.AI, stat.ML, deep learning, meta-learning, Machine learning, hyperparameter optimization, neural architecture search, automated machine learning, multi-fidelity optimization
ASJC Scopus subject areas
- Computer Science(all)
- Software
- Computer Science(all)
- Artificial Intelligence
- Mathematics(all)
- Applied Mathematics
- Computer Science(all)
- Computer Vision and Pattern Recognition
- Computer Science(all)
- Computational Theory and Mathematics
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
In: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, No. 9, 9382913, 01.09.2021, p. 3079-3090.
Research output: Contribution to journal › Article › Research › peer review
}
TY - JOUR
T1 - Auto-PyTorch
T2 - Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
AU - Zimmer, Lucas
AU - Lindauer, Marius
AU - Hutter, Frank
N1 - Funding Information: This work was supported in part by the Robert Bosch GmbH and in part by the European Research Council (ERC) under the European Unions Horizon 2020 Research and Innovation Program under Grant 716721.
PY - 2021/9/1
Y1 - 2021/9/1
N2 - While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search. In this paper, we introduce Auto-PyTorch, which brings the best of these two worlds together by jointly and robustly optimizing the architecture of networks and the training hyperparameters to enable fully automated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks by combining multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data. To thoroughly study our assumptions on how to design such an AutoDL system, we additionally introduce a new benchmark on learning curves for DNNs, dubbed LCBench, and run extensive ablation studies of the full Auto-PyTorch on typical AutoML benchmarks, eventually showing that Auto-PyTorch performs better than several state-of-the-art competitors on average.
AB - While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search. In this paper, we introduce Auto-PyTorch, which brings the best of these two worlds together by jointly and robustly optimizing the architecture of networks and the training hyperparameters to enable fully automated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks by combining multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data. To thoroughly study our assumptions on how to design such an AutoDL system, we additionally introduce a new benchmark on learning curves for DNNs, dubbed LCBench, and run extensive ablation studies of the full Auto-PyTorch on typical AutoML benchmarks, eventually showing that Auto-PyTorch performs better than several state-of-the-art competitors on average.
KW - cs.LG
KW - cs.AI
KW - stat.ML
KW - deep learning
KW - meta-learning
KW - Machine learning
KW - hyperparameter optimization
KW - neural architecture search
KW - automated machine learning
KW - multi-fidelity optimization
UR - http://www.scopus.com/inward/record.url?scp=85103276230&partnerID=8YFLogxK
U2 - 10.1109/TPAMI.2021.3067763
DO - 10.1109/TPAMI.2021.3067763
M3 - Article
VL - 43
SP - 3079
EP - 3090
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
JF - IEEE Transactions on Pattern Analysis and Machine Intelligence
SN - 0162-8828
IS - 9
M1 - 9382913
ER -