The auto selector and racing

The objective

The examples below use a shifted multimodal test function, a synthetic Rastrigin-style surface whose global minimum sits at 1.3 in each coordinate:

shifted <- function(x) {
  z <- x - 1.3
  sum(z^2 - 10 * cos(2 * pi * z) + 10)
}

Engines available on this build

Three sections below need an optional engine each, or a pair of them: GenSA for the noisy-objective rule and every race, cmaes alongside it for racing specifically, and nloptr (for nloptr_bobyqa) for the budget-sweep figure. Each guarded chunk checks for what it needs and prints one sentence in its place when an engine is missing.

has_gensa <- requireNamespace("GenSA", quietly = TRUE)
has_cmaes <- requireNamespace("cmaes", quietly = TRUE)
has_nloptr <- requireNamespace("nloptr", quietly = TRUE)
has_ggplot2 <- requireNamespace("ggplot2", quietly = TRUE)
race_ok <- has_gensa && has_cmaes
sweep_ok <- race_ok && has_nloptr && has_ggplot2

Tier 1: the rules

When the problem’s declaration already determines a good choice, auto applies a transparent rule and spends no evaluations on selection. A noisy objective goes to a noise-tolerant global engine. An expensive objective goes to Bayesian optimisation, which spends few true evaluations. A permutation design space goes to the native simulated annealer, described in Combinatorial optimisation and delta evaluation. And a small budget stays with the rules, because probing the landscape would eat too much of it. A noisy objective also never races – ranking engines on one short noisy trial each would rank them on luck – so its rule commits directly:

noisy_prob <- optim_problem(
  fn = function(x) sum(x^2) + rnorm(1, sd = 0.05),
  space = space_box(lower = c(-3, -3), upper = c(3, 3)),
  objective = objective_noisy(sd = 0.05),
  max_evals = 500, seed = 42
)
res_noisy <- optimix(noisy_prob)
res_noisy$provenance[c("engine", "why", "tier")]
#> $engine
#> [1] "gensa"
#> 
#> $why
#> [1] "noisy objective: gensa (noise-tolerant global)"
#> 
#> $tier
#> [1] 1

Tier 3: racing

For a deterministic objective with an ample budget, auto escalates from rules to evidence. It samples the landscape once with a Latin-hypercube design, gives each installed candidate engine a short trial from the best sampled point as a shared warm start, and commits the remaining budget to the trial leader, warm-started from the leader’s own best. The landscape sample and every trial evaluation are charged against the budget, and the final run receives only what is left.

set.seed(1)
res <- optimix(shifted, lower = c(-5, -5), upper = c(5, 5))
res
#> <optimix_result>
#>   optimiser : gensa
#>   value     : 0
#>   par       : 1.3, 1.3
#>   evals     : 2000
#>   converged : yes
res$provenance[c("engine", "tier", "why")]
#> $engine
#> [1] "gensa"
#> 
#> $tier
#> [1] 3
#> 
#> $why
#> [1] "race 3 engines (ELA quad R2 = 0.62); gensa led the race"
res$provenance$race
#>         gensa         cmaes nloptr_bobyqa 
#>   0.000000000   0.001694246   0.000000000

The provenance is how a result explains itself: the engine that ran (engine), why it was chosen (why), and which tier decided (tier is 1 for a rule and 3 for a race). A raced result also keeps the trial scores.

Asking for a race directly

method = "race" skips the deliberation and races the installed global engines immediately. It needs a continuous box space and at least two installed candidates:

set.seed(2)
raced <- optimix(shifted, lower = c(-5, -5), upper = c(5, 5),
                 method = "race")
raced$provenance$why
#> [1] "explicit race of 3 engines; gensa led the race"

Budgets are respected

max_evals caps the true objective evaluations – a soft cap, since an engine may finish the internal step it is in – and the reported count is the true total across every stage of the search. A budget of 200 is too small for a race here (the landscape sample alone would cost a quarter of it), so auto stays with the tier-1 rule; when the chunk below executes, capped$counts[["function"]] sits at or just above 200, confirming the cap was honoured rather than silently overrun:

set.seed(3)
capped <- optimix(shifted, lower = c(-5, -5), upper = c(5, 5),
                  max_evals = 200)
capped$provenance$tier
#> [1] 1
capped$counts[["function"]]
#> [1] 200

Why the race commits to a leader

A single race, run once, shows which engine won; it does not show how the three race candidates behave as the budget grows, which is the reason the tier-3 rule commits the remaining budget to the trial leader rather than splitting it evenly. The sweep below re-solves the same shifted surface with each of gensa, cmaes, and nloptr_bobyqa directly, at six budgets from 30 to 1000 evaluations, and records the best value each reaches:

sweep_budgets <- c(30L, 60L, 120L, 250L, 500L, 1000L)
sweep_engines <- c("gensa", "cmaes", "nloptr_bobyqa")
sweep_grid <- expand.grid(
  engine = sweep_engines, budget = sweep_budgets, stringsAsFactors = FALSE
)
s1 <- nrow(sweep_grid)
sweep_rows <- vector("list", s1)
for (i1 in seq_len(s1)) {
  prob_i <- optim_problem(
    fn = shifted, space = space_box(lower = c(-5, -5), upper = c(5, 5)),
    max_evals = sweep_grid$budget[i1], seed = 5
  )
  res_i <- optimix(prob_i, method = sweep_grid$engine[i1])
  sweep_rows[[i1]] <- data.frame(
    engine = sweep_grid$engine[i1], budget = sweep_grid$budget[i1],
    value = res_i$value, evals = res_i$counts[["function"]]
  )
} # ends i1, over the engine x budget grid
sweep_df <- do.call(rbind, sweep_rows)
sweep_df
#>           engine budget        value evals
#> 1          gensa     30 9.949591e-01    91
#> 2          cmaes     30 8.026175e+00    30
#> 3  nloptr_bobyqa     30 1.989918e+00    32
#> 4          gensa     60 9.949591e-01    91
#> 5          cmaes     60 5.441800e+00    60
#> 6  nloptr_bobyqa     60 1.989918e+00    40
#> 7          gensa    120 9.949591e-01   120
#> 8          cmaes    120 1.348760e+00   120
#> 9  nloptr_bobyqa    120 1.989918e+00    40
#> 10         gensa    250 9.949591e-01   250
#> 11         cmaes    250 9.954714e-01   246
#> 12 nloptr_bobyqa    250 1.989918e+00    40
#> 13         gensa    500 1.776357e-15   500
#> 14         cmaes    500 9.949591e-01   498
#> 15 nloptr_bobyqa    500 1.989918e+00    40
#> 16         gensa   1000 1.776357e-15  1000
#> 17         cmaes   1000 9.949591e-01   996
#> 18 nloptr_bobyqa   1000 1.989918e+00    40

ggplot2::ggplot(
  sweep_df, ggplot2::aes(x = evals, y = value, colour = engine)
) +
  ggplot2::geom_line() +
  ggplot2::geom_point(size = 2) +
  ggplot2::scale_colour_viridis_d(name = "engine") +
  ggplot2::labs(
    x = "true objective evaluations",
    y = "best value found",
    title = "Convergence of the three raced engines, by budget"
  ) +
  ggplot2::theme_minimal()

Best value found against the true evaluation budget, for the three engines the tier-3 race chooses between. nloptr_bobyqa (a local search) settles early and stops improving regardless of budget; cmaes and gensa keep improving as the budget grows, with gensa reaching the global optimum by budget 500. This is what a single race’s trial-then-commit step is racing for.

The evidence

The comparison is against zakharov, salomon, and bohachevsky – three functions held out from the rules auto was designed against – each run at 3 dimensions x 6 seeds x 2 budgets (n = 54 instances per budget). These figures come from the package’s own tier-2/tier-3 held-out validation study and are not recomputed in this vignette. On that held-out set auto beat the best fixed single engine (gensa) where it matters most: at a constrained budget of 300 evaluations per dimension it solved more instances (success 0.63 against 0.56) at better median quality and lower cost. At a generous budget of 1000 evaluations per dimension it matched the single-best engine’s quality (an equal-or-tighter median gap) with about forty per cent fewer evaluations, but its success rate trails single-best’s at that budget (0.685 against 0.759) – racing spends part of the budget on trials that a strong single global would have spent solving, which costs a little when nearly every instance already favours that one engine. The efficiency win is real; the generous-budget success rate is the price paid for it.

A feature-based shortcut – route problems that look smooth to a cheap local solver – was prototyped and rejected on the same held-out evidence: one function is multimodal yet looks perfectly quadratic to a coarse sample (a bowl hiding a cosine ripple), so no feature threshold routes it safely. Racing is feature-free, resolving such cases by actually trying the engines, and that is why the escalation is a race rather than a landscape heuristic.