TY - JOUR

T1 - Auto-PyTorch

T2 - Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

AU - Zimmer, Lucas

AU - Lindauer, Marius

AU - Hutter, Frank

N1 - Funding Information: This work was supported in part by the Robert Bosch GmbH and in part by the European Research Council (ERC) under the European Unions Horizon 2020 Research and Innovation Program under Grant 716721.

PY - 2021/9/1

Y1 - 2021/9/1

N2 - While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search. In this paper, we introduce Auto-PyTorch, which brings the best of these two worlds together by jointly and robustly optimizing the architecture of networks and the training hyperparameters to enable fully automated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks by combining multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data. To thoroughly study our assumptions on how to design such an AutoDL system, we additionally introduce a new benchmark on learning curves for DNNs, dubbed LCBench, and run extensive ablation studies of the full Auto-PyTorch on typical AutoML benchmarks, eventually showing that Auto-PyTorch performs better than several state-of-the-art competitors on average.

AB - While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search. In this paper, we introduce Auto-PyTorch, which brings the best of these two worlds together by jointly and robustly optimizing the architecture of networks and the training hyperparameters to enable fully automated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks by combining multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data. To thoroughly study our assumptions on how to design such an AutoDL system, we additionally introduce a new benchmark on learning curves for DNNs, dubbed LCBench, and run extensive ablation studies of the full Auto-PyTorch on typical AutoML benchmarks, eventually showing that Auto-PyTorch performs better than several state-of-the-art competitors on average.

KW - cs.LG

KW - cs.AI

KW - stat.ML

KW - deep learning

KW - meta-learning

KW - Machine learning

KW - hyperparameter optimization

KW - neural architecture search

KW - automated machine learning

KW - multi-fidelity optimization

UR - http://www.scopus.com/inward/record.url?scp=85103276230&partnerID=8YFLogxK

U2 - 10.1109/TPAMI.2021.3067763

DO - 10.1109/TPAMI.2021.3067763

M3 - Article

VL - 43

SP - 3079

EP - 3090

JO - IEEE Transactions on Pattern Analysis and Machine Intelligence

JF - IEEE Transactions on Pattern Analysis and Machine Intelligence

SN - 0162-8828

IS - 9

M1 - 9382913

ER -