TY - GEN

T1 - SynthACticBench: A Capability-Based Synthetic Benchmark for Algorithm Configuration

AU - Margraf, Valentin

AU - Lappe, Anna

AU - Wever, Marcel Dominik

AU - Benjamins, Carolin

AU - Hüllermeier, Eyke

AU - Lindauer, Marius

PY - 2025/6

Y1 - 2025/6

N2 - Algorithm configuration deals with the automatic optimization of an algorithm's parameters to maximize its performance on a distribution of problem instances, such as Boolean satisfiability or the traveling salesperson problem. While significant progress has been made in developing optimizers for algorithm configuration -- so-called algorithm configurators -- their evaluation remains computationally expensive and often relies on real-world scenarios with hard-to-control characteristics. This makes it challenging to analyze their strengths and weaknesses systematically. To address this, we introduce SynthACticBench, a synthetic benchmark specifically designed to isolate and investigate key properties of algorithm configuration problems. Our benchmark distinguishes between properties related to the configuration space and those associated with the objective function. We define a configurator's ability to handle a particular property as its capability -- for example, the capability to manage hierarchical configuration spaces. Using SynthACticBench, we evaluate two state-of-the-art algorithm configurators, SMAC and irace, examining their complementary capabilities and analyzing their performances across diverse benchmark functions. By providing a controlled, scalable, and capability-based evaluation environment, SynthACticBench facilitates a more targeted analysis of algorithm configurators, helping to advance research in the field. The benchmark is available at: https://github.com/annaelisalappe/SynthACticBench.

AB - Algorithm configuration deals with the automatic optimization of an algorithm's parameters to maximize its performance on a distribution of problem instances, such as Boolean satisfiability or the traveling salesperson problem. While significant progress has been made in developing optimizers for algorithm configuration -- so-called algorithm configurators -- their evaluation remains computationally expensive and often relies on real-world scenarios with hard-to-control characteristics. This makes it challenging to analyze their strengths and weaknesses systematically. To address this, we introduce SynthACticBench, a synthetic benchmark specifically designed to isolate and investigate key properties of algorithm configuration problems. Our benchmark distinguishes between properties related to the configuration space and those associated with the objective function. We define a configurator's ability to handle a particular property as its capability -- for example, the capability to manage hierarchical configuration spaces. Using SynthACticBench, we evaluate two state-of-the-art algorithm configurators, SMAC and irace, examining their complementary capabilities and analyzing their performances across diverse benchmark functions. By providing a controlled, scalable, and capability-based evaluation environment, SynthACticBench facilitates a more targeted analysis of algorithm configurators, helping to advance research in the field. The benchmark is available at: https://github.com/annaelisalappe/SynthACticBench.

KW - AutoML

KW - Parameter tuning and algorithm configuration

KW - Benchmark problems

KW - Reproducibility aspects

M3 - Conference contribution

T3 - ACM Conferences

BT - GECCO 2025 - Proceedings of the 2025 Genetic and Evolutionary Computation Conference

PB - Association for Computing Machinery (ACM)

ER -