Limits and honesty

Two sets of numbers on this page

The primary evidence is the preregistered final test (FT1): the frozen configuration run once on bbob instance ids 101-110, which no design decision used, with every competitor at every dimension it was planned for (NGOpt at all dimensions (v2)). The development context further down uses ids 61-70, which were used while AMM was being built (in-sample). The two use different rules for a “behind” cell and their counts must not be compared with each other.

This page says what AMM does not do well, what it was and was not tested on, and what the project does not claim. It is meant to be read before the results, not after.

1. In the final test (ids 101-110)

The frozen configuration (preset c46, config hash 2153948ebed51150) was run once on bbob instance ids 101-110: 24 functions, dimensions 2, 5, 10, 20 and 40, budgets of 10, 30, 100, 300 and 1000 evaluations per dimension, ten instances per cell, one run per cell and instance. That is 600 cells and 6,000 runs of AMM. The field is pycma IPOP and BIPOP, restarted SLSQP, SciPy differential evolution, Optuna’s TPE (a capped slice of the grid), Nevergrad’s NGOpt and HCMA as COCO archive levels (its own instances, D 2 to 20 only, labelled as such).

Credibly behind: the rule

A cell counts as credibly behind if some competitor leads AMM by more than 0.02 in the share of targets reached with posterior probability above 0.9 over the ten paired runs. This is the preregistered rule F2, taken from the project’s field ledger. It is not the plotting rule of the development section below. Cell-level intervals of this kind were found to be about 25% too narrow in earlier work, so the counts are counts and not significance statements.

35cells where AMM is credibly behind some competitor, of 600 (final test)

That is 5.8% of the cells. In 110 cells AMM is credibly ahead of every competitor that ran the cell, and in 382 cells (“64%”) it has the highest mean share of targets, ties included. The three counts answer different questions and overlap only in part: a cell can be neither credibly behind nor credibly ahead.

The cells, listed

All 35 cells are listed below with the competitor that leads in each. The tables are built from the final-test data file, and a build fails if they disagree with the counts quoted on this site.

final test FT1, NGOpt at every dimension35 of 600 cells where AMM (frozen configuration c46) is credibly behind some competitor, on bbob ids 101-110. One run per cell and instance, ten instances per cell.

By dimension and budget

Credibly behind cells by dimension and budget (evaluations per dimension). Each entry counts the functions, out of 24, in that cell of the grid.

Dimension	10 x D	30 x D	100 x D	300 x D	1000 x D	All
D = 2	4	2	0	3	2	11
D = 5	2	3	1	1	0	7
D = 10	2	3	2	1	0	8
D = 20	1	1	0	1	1	4
D = 40	2	1	1	0	1	5
All dimensions	11	10	4	6	4	35

By competitor

For each competitor: cells where it is among the credible leaders over AMM, cells where AMM is credibly ahead of it, and the cells it ran. A cell with a tie between competitors counts for each of them.

Competitor	Credibly behind it	Credibly ahead of it	Cells it ran
SLSQP	11	299	600
HCMA (archive) (D 2-20, its own instances)	11	173	480
pycma IPOP	6	323	600
pycma BIPOP	7	326	600
TPE (Optuna) (capped slice of the grid)	2	149	288
NGOpt (Nevergrad)	6	346	600
DE	0	455	600

All 35 cells, grouped by function

Every cell where AMM is credibly behind some competitor. Delta is AMM minus the leading competitor in the mean share of the 51 targets reached (ten instances); it is negative in every row. Where a row names several competitors, they have the same delta in the data.

Function	D	Budget	Delta	Leading competitor
f2Ellipsoidal, separable	2	10 x D	−0.096	SLSQP
	2	30 x D	−0.086	SLSQP
f7Step ellipsoidal	10	100 x D	−0.076	HCMA (archive)
	20	300 x D	−0.267	HCMA (archive)
f8Rosenbrock	10	30 x D	−0.069	SLSQP
f9Rosenbrock, rotated	5	10 x D	−0.065	NGOpt (Nevergrad)
	10	10 x D	−0.047	NGOpt (Nevergrad)
f10Ellipsoidal	5	30 x D	−0.310	SLSQP
	10	30 x D	−0.178	SLSQP
f11Discus	5	10 x D	−0.078	SLSQP
f12Bent cigar	2	10 x D	−0.208	SLSQP
	2	30 x D	−0.259	SLSQP
	2	300 x D	−0.143	pycma IPOPpycma BIPOP
f13Sharp ridge	2	10 x D	−0.065	SLSQP
	2	300 x D	−0.078	HCMA (archive)
	20	10 x D	−0.029	HCMA (archive)
	40	100 x D	−0.025	SLSQP
f14Different powers	2	10 x D	−0.057	SLSQP
	2	300 x D	−0.122	pycma IPOPpycma BIPOPHCMA (archive)
	20	1000 x D	−0.037	pycma IPOPpycma BIPOPHCMA (archive)
	40	1000 x D	−0.041	pycma IPOPpycma BIPOP
f17Schaffers F7	5	30 x D	−0.059	TPE (Optuna)
	5	100 x D	−0.065	NGOpt (Nevergrad)
	5	300 x D	−0.090	NGOpt (Nevergrad)
	40	30 x D	−0.025	pycma IPOPpycma BIPOP
f18Schaffers F7, ill-conditioned	2	1000 x D	−0.153	pycma BIPOP
	5	30 x D	−0.043	TPE (Optuna)
	20	30 x D	−0.033	pycma IPOPpycma BIPOP
f19Griewank-Rosenbrock	10	10 x D	−0.028	HCMA (archive)
	10	30 x D	−0.054	HCMA (archive)
	10	100 x D	−0.031	HCMA (archive)
	10	300 x D	−0.026	HCMA (archive)
f20Schwefel	40	10 x D	−0.035	NGOpt (Nevergrad)
f23Katsuura	40	10 x D	−0.031	NGOpt (Nevergrad)
f24Lunacek bi-Rastrigin	2	1000 x D	−0.037	HCMA (archive)

Read with care. "Credibly behind" is the preregistered rule: a competitor leads by more than 0.02 in target share with posterior probability above 0.9 over the ten paired runs. Cell-level intervals of this kind were found to be about 25% too narrow in earlier work, so the counts on this page are counts and not significance statements. NGOpt ran at every dimension. HCMA is not run in this harness: its values are the COCO archive's, on its own instances, for D 2 to 20, and TPE ran a capped slice of the grid. Where a competitor did not run a cell it cannot lead it.

Source: data/ft1/ft1_v2.json, from the analysis script final_ft1.py run with no arguments after every arm had finished.

The pattern, by type

Read from the list above. Each type names functions and conditions; the table gives the deltas. A cell where several competitors have the same delta is counted for each of them, so the per-competitor counts add up to more than 35.

• Restarted SLSQP, 11 cells. Small budgets on smooth functions: f2, f8, f10, f11, f12, f13 and f14, at 10 to 30 evaluations per dimension, in dimensions 2 to 10. The largest gaps of the whole list are in this group: f10 at D 5 with 30 x D, and f12 at D 2 with 30 x D. One SLSQP cell sits outside the pattern: f13 at D 40 and 100 x D, with a small gap.
• HCMA’s archive levels, 11 cells. On f7 at D 10 (100 x D) and D 20 (300 x D); the second is one of the larger gaps in the list. On f19 at D 10 at every budget from 10 to 300 x D, with small gaps. On f13 at D 2 with 300 x D and at D 20 with 10 x D, on f24 at D 2 with 1000 x D, and on f14 at D 2 (300 x D) and D 20 (1000 x D), where IPOP and BIPOP have the same delta. HCMA values come from the COCO archive on other instances, so these cells say that AMM is below an archive level, not that HCMA wins a paired run.
• IPOP, 6 cells, and BIPOP, 7 cells. Almost always together. Larger gaps at D 2 on f12 and f14 (300 x D); BIPOP alone on f18 at D 2 (1000 x D). Small gaps on f14 at D 20 and 40 (1000 x D) and on f17 and f18 at D 20 to 40 (30 x D).
• NGOpt, 6 cells. Two are on f9 at 10 x D (D 5 and D 10), two on f17 at D 5 (100 x D and 300 x D), one on f20 at D 40 (10 x D) and one on f23 at D 40 (10 x D). None of these gaps is among the largest of the list; the largest is f17 at D 5 with 300 x D. None of the six is at D 2 or D 20.
• TPE, 2 cells. f17 and f18 at D 5 with 30 x D.
• Differential evolution, 0 cells. It leads in none.

By dimension the counts are 11, 7, 8, 4 and 5 at D 2, 5, 10, 20 and 40. By budget they are 11, 10, 4, 6 and 4 at 10, 30, 100, 300 and 1000 x D. Over the full grid, AMM is credibly ahead of every competitor that ran the cell in 12, 21, 19, 20 and 38 cells by dimension, and in 2, 12, 33, 32 and 31 cells by budget. The losses are where development had them: small budgets on smooth functions against restarted SLSQP, f7 and f19 against HCMA’s levels, a few D 2 cells against IPOP and BIPOP at 300 to 1000 x D, and a handful of small-budget cells against NGOpt and TPE. Fewer cells are lost at 100 x D and above than at 10 and 30 x D, and fewer at D 20 and 40 than at D 2. This is a reading of a list of counts, not a test of a cause.

What was predicted

The count of behind cells and the count of cells ahead of all were preregistered as ranges before any run. The falsifiers were also written down: more than 100 behind cells, or fewer than 50 ahead of all, would have meant that the development ledger was optimistic beyond the stated ranges.

Quantity	Predicted (preregistered)	Final test	Reading
Cells credibly behind some competitor (F2)	30 to 75 of 600	35	Inside the range. Falsifier (above 100) not triggered.
Cells credibly ahead of every competitor that ran the cell (F3)	70 to 120 of 600	110	Inside the range. Falsifier (below 50) not triggered.

Both counts are inside their ranges, and so are the ten comparisons with IPOP and SLSQP (F1, F4), which are on the research log. Every preregistered prediction of the final test held. The research log records how the analysis was run and the deviations from the plan.

The final test confirms the direction of the development result and does not prove that AMM wins on other problems: it is still the same 24 functions, on new instances. See “What is not claimed” below.

2. Development context, in-sample ids 61-70

Development data, not the final test

Everything in this section is in-sample: bbob instance ids 61-70 were used while AMM was being built, and steered about twenty composites. It uses the plotting rule below on one seed per instance, not the preregistered “credibly behind” rule of section 1. Do not compare the two counts.

In the in-sample comparison of the frozen configuration (preset c46) against pycma IPOP and BIPOP, SLSQP, differential evolution, Optuna’s TPE, Nevergrad’s NGOpt and HCMA, there are 600 cells, each one function at one dimension and one budget. The rule is fixed:

• ahead if AMM’s share of targets reached exceeds the best competitor’s by more than 0.02: 179in-sample cells;
• level if the difference is within 0.02: 313in-sample cells;
• behind if the best competitor leads by more than 0.02: 108in-sample cells.

One seed per instance and ten instances per cell: there is no interval, so 0.02 is a plotting threshold and not a significance test. This is not the preregistered “credibly behind” count of section 1, which uses a posterior probability over the paired runs; the two counts are different and are not mixed here. The same cells appear on the results heatmap under this plotting rule.

108in-samplecells where AMM is behind the best competitor by more than 0.02 (in-sample)

AMM (c46)pycma IPOPpycma BIPOPSLSQPDETPE (Optuna)NGOpt (Nevergrad)HCMA (COCO archive)

The pattern in the behind cells

The tables below list every behind cell. Reading them, in order of how clearly the data show it:

• Restarted SLSQP is the most frequent leader. It leads where AMM is behind more often than any other competitor, mostly at 10 to 100 evaluations per dimension, mainly on the unimodal functions with moderate or high conditioning (f8 to f13) and on Gallagher’s f21 and f22. The paper’s development record found the same pattern early, when SLSQP beat an early composite on smooth functions. This table does not establish the cause.
• Small budgets lose more cells than large ones. More behind cells sit at 10 and 30 times the dimension than at 300 and 1000. Nevergrad’s NGOpt leads mostly at 10 and 30 times the dimension; IPOP, BIPOP and differential evolution lead mostly at 100 times the dimension and above.
• Dimension 2 has the most behind cells, and D = 40 the fewest.
• Function groups. Few separable-function cells (f1 to f5) are behind. The rest are spread over the moderate- and high-conditioning groups, the adequately structured multimodal group and the weakly structured multimodal group. The functions with the most behind cells are f8 and f9 (Rosenbrock, unrotated and rotated), then f7 (step ellipsoid), and f10, f12, f13, f17 and f21.
• HCMA’s leads are against archive data. HCMA is not run in this harness: its values are the COCO archive’s, on other instances, and cover D 2 to 20 only. A cell where HCMA leads says that AMM is below an archive level, not that HCMA beats AMM in a paired run.
• The paper’s own section on losses describes an earlier configuration. The section “Where AMM does not win” in the paper draft was written for composite c43. Its largest gaps (f7 at D = 5 and 100 times the dimension, against IPOP; f9 at D = 10 and 300 times the dimension, against SLSQP) have changed since: the f9 cell is unchanged and still listed below, while the f7 cell is much smaller. The list in section 1 is the one to use.

development, in-sample, ids 61-70108 of 600 cells where AMM (frozen configuration c46) is behind the best competitor by more than 0.02 in the share of targets reached. One seed per instance, ten instances per cell.

By the competitor that leads

Behind cells by the leading competitor and by budget (evaluations per dimension)

Leading competitor	All	10 x D	30 x D	100 x D	300 x D	1000 x D
SLSQP	40	9	13	10	5	3
NGOpt (Nevergrad)	22	10	7	4	1	0
HCMA (archive)	18	4	2	3	6	3
pycma IPOP	14	0	3	4	3	4
TPE (Optuna)	8	2	6	0	0	0
DE	4	0	1	0	1	2
pycma BIPOP	2	0	0	0	0	2
All competitors	108	25	32	21	16	14

By function and dimension

Behind cells per function and dimension. Each entry counts the budgets, out of 5, at which AMM is behind; shading follows the count.

Function	D = 2	D = 5	D = 10	D = 20	D = 40	All
Group 1: separable (f1 to f5)
f1Sphere	0	0	0	1	0	1
f2Ellipsoidal, separable	1	1	0	0	0	2
f3Rastrigin, separable	1	0	0	0	0	1
f4Bueche-Rastrigin	0	0	0	0	0	0
f5Linear slope	0	0	0	0	0	0
Group 2: low or moderate conditioning (f6 to f9)
f6Attractive sector	0	1	0	0	1	2
f7Step ellipsoidal	3	1	3	1	0	8
f8Rosenbrock	1	2	2	4	1	10
f9Rosenbrock, rotated	0	1	3	3	3	10
Group 3: high conditioning, unimodal (f10 to f14)
f10Ellipsoidal	3	2	1	0	1	7
f11Discus	3	1	0	1	0	5
f12Bent cigar	3	1	1	1	1	7
f13Sharp ridge	2	3	1	1	0	7
f14Different powers	2	0	0	0	1	3
Group 4: multimodal, adequate global structure (f15 to f19)
f15Rastrigin	2	0	0	0	0	2
f16Weierstrass	3	3	0	0	0	6
f17Schaffers F7	3	2	0	1	1	7
f18Schaffers F7, ill-conditioned	2	2	0	1	1	6
f19Griewank-Rosenbrock	0	1	3	1	0	5
Group 5: multimodal, weak global structure (f20 to f24)
f20Schwefel	0	0	0	0	1	1
f21Gallagher, 101 peaks	2	2	0	2	1	7
f22Gallagher, 21 peaks	0	0	1	2	0	3
f23Katsuura	1	0	0	1	2	4
f24Lunacek bi-Rastrigin	2	1	1	0	0	4
All functions	34	24	16	20	14	108

All 108 cells, one row each

Every cell where AMM is behind by more than 0.02. Values are the share of the 51 targets reached (0 to 1), mean over ten instances. Delta is AMM minus the best competitor.

Function	D	Budget	AMM	Best competitor	Its value	Delta	No data for
f1Sphere	20	10 x D	0.967	SLSQP	1.000	−0.033	none
f2Ellipsoidal, separable	2	10 x D	0.110	SLSQP	0.161	−0.051	none
	5	30 x D	0.720	SLSQP	0.802	−0.082	none
f3Rastrigin, separable	2	100 x D	0.286	HCMA (archive)	0.505	−0.218	none
f6Attractive sector	5	10 x D	0.063	NGOpt (Nevergrad)	0.094	−0.031	none
	40	100 x D	0.049	pycma IPOP	0.075	−0.025	TPE (Optuna), HCMA (archive)
f7Step ellipsoidal	2	30 x D	0.182	TPE (Optuna)	0.263	−0.080	none
	2	100 x D	0.480	pycma IPOP	0.551	−0.071	none
	2	300 x D	0.943	DE	0.965	−0.022	TPE (Optuna)
	5	100 x D	0.514	pycma IPOP	0.612	−0.098	none
	10	10 x D	0.043	HCMA (archive)	0.065	−0.022	NGOpt (Nevergrad)
	10	100 x D	0.186	HCMA (archive)	0.264	−0.078	NGOpt (Nevergrad)
	10	300 x D	0.637	HCMA (archive)	0.715	−0.078	TPE (Optuna), NGOpt (Nevergrad)
	20	300 x D	0.282	HCMA (archive)	0.556	−0.273	TPE (Optuna)
f8Rosenbrock	2	10 x D	0.214	NGOpt (Nevergrad)	0.241	−0.027	none
	5	100 x D	0.778	SLSQP	0.916	−0.137	none
	5	300 x D	0.835	SLSQP	1.000	−0.165	TPE (Optuna)
	10	30 x D	0.100	SLSQP	0.155	−0.055	NGOpt (Nevergrad)
	10	300 x D	0.914	SLSQP	1.000	−0.086	TPE (Optuna), NGOpt (Nevergrad)
	20	10 x D	0.000	NGOpt (Nevergrad)	0.022	−0.022	none
	20	100 x D	0.231	SLSQP	0.324	−0.092	TPE (Optuna)
	20	300 x D	0.831	HCMA (archive)	0.976	−0.145	TPE (Optuna)
	20	1000 x D	0.839	pycma IPOP	1.000	−0.161	TPE (Optuna)
	40	30 x D	0.002	NGOpt (Nevergrad)	0.029	−0.027	TPE (Optuna), HCMA (archive)
f9Rosenbrock, rotated	5	10 x D	0.080	NGOpt (Nevergrad)	0.131	−0.051	none
	10	30 x D	0.106	SLSQP	0.133	−0.027	NGOpt (Nevergrad)
	10	100 x D	0.702	SLSQP	0.743	−0.041	NGOpt (Nevergrad)
	10	300 x D	0.663	SLSQP	1.000	−0.337	TPE (Optuna), NGOpt (Nevergrad)
	20	10 x D	0.000	NGOpt (Nevergrad)	0.024	−0.024	none
	20	30 x D	0.033	NGOpt (Nevergrad)	0.061	−0.027	none
	20	100 x D	0.192	SLSQP	0.241	−0.049	TPE (Optuna)
	40	30 x D	0.008	NGOpt (Nevergrad)	0.051	−0.043	TPE (Optuna), HCMA (archive)
	40	300 x D	0.551	SLSQP	0.827	−0.276	TPE (Optuna), HCMA (archive)
	40	1000 x D	0.914	SLSQP	1.000	−0.086	TPE (Optuna), HCMA (archive)
f10Ellipsoidal	2	10 x D	0.120	SLSQP	0.241	−0.122	none
	2	30 x D	0.706	SLSQP	0.763	−0.057	none
	2	300 x D	0.941	HCMA (archive)	1.000	−0.059	TPE (Optuna)
	5	10 x D	0.000	SLSQP	0.035	−0.035	none
	5	30 x D	0.563	SLSQP	0.759	−0.196	none
	10	30 x D	0.114	SLSQP	0.167	−0.053	NGOpt (Nevergrad)
	40	100 x D	0.394	SLSQP	0.451	−0.057	TPE (Optuna), HCMA (archive)
f11Discus	2	10 x D	0.100	SLSQP	0.175	−0.075	none
	2	30 x D	0.749	SLSQP	0.798	−0.049	none
	2	300 x D	0.955	HCMA (archive)	1.000	−0.045	TPE (Optuna)
	5	10 x D	0.169	SLSQP	0.288	−0.120	none
	20	10 x D	0.182	SLSQP	0.225	−0.043	none
f12Bent cigar	2	30 x D	0.578	SLSQP	0.780	−0.202	none
	2	100 x D	0.708	SLSQP	0.875	−0.167	none
	2	300 x D	0.845	SLSQP	0.955	−0.110	TPE (Optuna)
	5	30 x D	0.039	NGOpt (Nevergrad)	0.096	−0.057	none
	10	1000 x D	0.918	HCMA (archive)	1.000	−0.082	TPE (Optuna), NGOpt (Nevergrad)
	20	30 x D	0.063	NGOpt (Nevergrad)	0.102	−0.039	none
	40	30 x D	0.071	NGOpt (Nevergrad)	0.145	−0.075	TPE (Optuna), HCMA (archive)
f13Sharp ridge	2	10 x D	0.131	SLSQP	0.198	−0.067	none
	2	30 x D	0.622	SLSQP	0.700	−0.078	none
	5	10 x D	0.055	NGOpt (Nevergrad)	0.080	−0.025	none
	5	30 x D	0.245	SLSQP	0.271	−0.025	none
	5	1000 x D	0.955	pycma IPOP	1.000	−0.045	TPE (Optuna)
	10	300 x D	0.918	HCMA (archive)	1.000	−0.082	TPE (Optuna), NGOpt (Nevergrad)
	20	10 x D	0.022	HCMA (archive)	0.044	−0.023	none
f14Different powers	2	30 x D	0.714	SLSQP	0.784	−0.071	none
	2	300 x D	0.882	pycma IPOP	1.000	−0.118	TPE (Optuna)
	40	1000 x D	0.861	pycma IPOP	0.910	−0.049	TPE (Optuna), HCMA (archive)
f15Rastrigin	2	100 x D	0.206	NGOpt (Nevergrad)	0.233	−0.027	none
	2	1000 x D	0.580	DE	0.843	−0.263	TPE (Optuna)
f16Weierstrass	2	30 x D	0.163	DE	0.194	−0.031	none
	2	300 x D	0.543	pycma IPOP	0.635	−0.092	TPE (Optuna)
	2	1000 x D	0.898	DE	1.000	−0.102	TPE (Optuna)
	5	30 x D	0.106	TPE (Optuna)	0.129	−0.024	none
	5	100 x D	0.159	NGOpt (Nevergrad)	0.190	−0.031	none
	5	1000 x D	0.512	pycma BIPOP	0.671	−0.159	TPE (Optuna)
f17Schaffers F7	2	10 x D	0.145	TPE (Optuna)	0.188	−0.043	none
	2	30 x D	0.194	TPE (Optuna)	0.257	−0.063	none
	2	1000 x D	0.888	pycma BIPOP	0.949	−0.061	TPE (Optuna)
	5	30 x D	0.173	TPE (Optuna)	0.216	−0.043	none
	5	300 x D	0.427	NGOpt (Nevergrad)	0.463	−0.035	TPE (Optuna)
	20	30 x D	0.131	pycma IPOP	0.155	−0.024	none
	40	30 x D	0.118	pycma IPOP	0.153	−0.035	TPE (Optuna), HCMA (archive)
f18Schaffers F7, ill-conditioned	2	30 x D	0.125	TPE (Optuna)	0.171	−0.045	none
	2	100 x D	0.196	pycma IPOP	0.251	−0.055	none
	5	30 x D	0.104	TPE (Optuna)	0.155	−0.051	none
	5	100 x D	0.212	NGOpt (Nevergrad)	0.243	−0.031	none
	20	100 x D	0.178	NGOpt (Nevergrad)	0.208	−0.029	TPE (Optuna)
	40	30 x D	0.063	pycma IPOP	0.086	−0.024	TPE (Optuna), HCMA (archive)
f19Griewank-Rosenbrock	5	30 x D	0.159	HCMA (archive)	0.195	−0.036	none
	10	10 x D	0.129	HCMA (archive)	0.152	−0.022	NGOpt (Nevergrad)
	10	30 x D	0.137	HCMA (archive)	0.186	−0.048	NGOpt (Nevergrad)
	10	100 x D	0.153	HCMA (archive)	0.188	−0.035	NGOpt (Nevergrad)
	20	1000 x D	0.186	pycma IPOP	0.214	−0.027	TPE (Optuna)
f20Schwefel	40	10 x D	0.120	NGOpt (Nevergrad)	0.159	−0.039	HCMA (archive)
f21Gallagher, 101 peaks	2	10 x D	0.220	SLSQP	0.296	−0.076	none
	2	30 x D	0.247	SLSQP	0.445	−0.198	none
	5	10 x D	0.167	NGOpt (Nevergrad)	0.194	−0.027	none
	5	30 x D	0.151	NGOpt (Nevergrad)	0.280	−0.129	none
	20	10 x D	0.139	HCMA (archive)	0.161	−0.022	none
	20	100 x D	0.184	SLSQP	0.431	−0.247	TPE (Optuna)
	40	100 x D	0.227	SLSQP	0.267	−0.039	TPE (Optuna), HCMA (archive)
f22Gallagher, 21 peaks	10	100 x D	0.175	SLSQP	0.351	−0.176	NGOpt (Nevergrad)
	20	10 x D	0.100	NGOpt (Nevergrad)	0.124	−0.024	none
	20	100 x D	0.151	SLSQP	0.239	−0.088	TPE (Optuna)
f23Katsuura	2	30 x D	0.169	SLSQP	0.204	−0.035	none
	20	1000 x D	0.210	SLSQP	0.235	−0.025	TPE (Optuna)
	40	10 x D	0.133	NGOpt (Nevergrad)	0.157	−0.024	HCMA (archive)
	40	1000 x D	0.210	SLSQP	0.235	−0.025	TPE (Optuna), HCMA (archive)
f24Lunacek bi-Rastrigin	2	10 x D	0.110	TPE (Optuna)	0.133	−0.024	none
	2	1000 x D	0.190	HCMA (archive)	0.233	−0.042	TPE (Optuna)
	5	1000 x D	0.120	HCMA (archive)	0.140	−0.020	TPE (Optuna)
	10	300 x D	0.051	pycma IPOP	0.075	−0.024	TPE (Optuna), NGOpt (Nevergrad)

Read with care. Where a competitor has no data for a cell, the best competitor is the best of the others, so those cells are easier to win and a missing competitor can hide a loss. HCMA has no D = 40 values and comes from the COCO archive, on other instances. TPE ran a capped slice only. NGOpt has no D = 10 runs. The last column of the full list names the competitors with no data in each cell.

The margin of 0.02 is a plotting threshold, not a significance test: there is one seed per instance and no interval. Source: data/dev/field_cells.json, generated from the c46 field plots.

3. Problem classes the paper flags as risks

These come from the paper’s discussion of limitations. The final test measures bbob only, so none of them has been measured on it.

• Rotated separable problems. Probes that read coordinate structure (the pair test, the coordinate-search route, STEP restarts) cannot fire on a rotated separable function. AMM falls back to its general path and has paid for the probe.
• Anytime use. AMM chooses its route from the budget, so a run stopped early returns the incumbent of a schedule planned for the full budget. An anytime measure can rank AMM below its final-budget value. COCO’s anytime ECDF represents it only with one dataset per budget.
• Cheap objectives. AMM is written in Python. Overhead per evaluation matters when the objective is fast and the dimension is small. The only stored figure is for SciPy’s COBYQA alone, about 4.4 ms per evaluation at D 2 to 3, measured outside COCO’s procedure. The figure for AMM as a whole is pendingpending.
• Probes outside their calibrated range. In development, a separability route calibrated at 300 to 1000 evaluations per dimension misrouted rugged functions at 3000, with a loss of −.18 to −.25 per cell, and a fractal exemption lost .16 on f16. Validity ranges limit such failures; they cannot rule out a misreading on a problem class unlike bbob’s.
• Absolute constants. The kink threshold, the restart tolerance and the finite-difference step are absolute, and the probe scales are fixed fractions of the box width. Several probes are invariant to a positive affine map of f; the constants are not.

4. Intended use and out of scope

A model-card summary. “Tested” means exercised in the benchmark runs behind this site; “untested” means nothing was run and no claim is made either way.

Intended use	Minimising a noiseless, continuous objective in a box, with 2 to 40 variables, when the number of evaluations is known before the run and lies between 10 and 1000 times the number of variables.
Needs	Exact function values (finite differences and ties rely on them), a box (bounds on every variable), a start point and a budget, and a seed for reproducibility.
Tested on	The 24 noiseless bbob functions, five dimensions (2, 5, 10, 20, 40), five budgets (10, 30, 100, 300, 1000 times the dimension). The final test (ids 101-110) covers the same grid. The paper’s plan also covers 3 variables and 3000 times the dimension; those results are pending.
Not an anytime method	The route depends on the budget. Stopping a run early gives the incumbent of a plan made for the full budget.
Better tools exist when	Gradients are cheap (use a gradient-based method), the budget is a handful of evaluations per variable with a smooth function (a restarted quasi-Newton method leads in some of the cells above), or each evaluation is so cheap that Python overhead dominates.
Out of scope, untested	Noisy objectives, discrete or mixed-integer variables, constraints other than the box, multi-objective problems, more than 40 variables, budgets below 10 or above 1000 times the dimension (3000 pending), objectives that fail or return infinities as a rule.
Beyond bbob	not yet testedpending. MA-BBOB: pendingpending. Transformed bbob and real problems are not yet tested.

5. What is not claimed

Taken from the paper’s claim inventory: rows with status N (not supportable) or G (supportable only after a named gap closes). The site’s own claim file marks any pending number, and a public build refuses to publish a pending one.

Statement	Status	Why, and what would change it
AMM beats every competitor in every cell, or is state of the art on bbob	Not claimed	The table above lists the cells where it does not. The paper does not use this wording.
The ATM sampler drives AMM’s gains	Not claimed	The paper argues the opposite: the routing, not the sampler, is the contribution. The sampler runs only above 100 times the dimension.
AMM leads the field on never-used official instances	Tested once	The preregistered final test FT1 on ids 101-110: AMM has the highest mean in 382 of 600 cells (ties included) and is credibly behind some competitor in 35, all listed in section 1. This is bbob only, on the same 24 functions as the development, with new instances.
The in-sample cell counts are a result	Not claimed	They come from ids that steered about twenty composites. They are development context only; the final test of section 1 is the evidence.
The final-test counts F2 and F3 are significance statements	Not claimed	They are counts under a preregistered rule whose cell-level intervals are about 25% too narrow. The comparisons with IPOP and SLSQP (F1, F4) carry the intervals.
AMM generalises beyond bbob’s function identities	Pending	Needs MA-BBOB, transformed bbob and other problems: not yet testedpending.
Every route (each probe-and-engine pair) pays its way in the final configuration	Pending	Needs the leave-one-route-out ablation: pendingpending. Earlier confirmations are add-one results against a predecessor on a target slice and do not measure the marginal value in the final configuration.
The routing, not the solver pool, carries the gain	Pending	Needs the ladder of controls: each engine alone, fixed hand-overs, NgIohTuned, a greedy schedule of AMM’s own engines.
AMM beats a simple SLSQP-then-IPOP portfolio built from its own parts	Pending	Part of the same ladder.
Routes fire on landscape properties and not on function fingerprints	Pending	Needs a firing table. The paper concedes that the chain route can be read as an f20 device.
AMM is ahead of HCMA from some budget upward	Descriptive, FT1	Against archive levels (D 2 to 20, other instances), not a paired run. AMM minus HCMA’s level is +.120 at 30 x D, +.174 at 100, +.104 at 300 and +.103 at 1000 x D. At 10 x D the margin is small (+.016), so the claim is stated from 30 x D.
CPU cost per evaluation	Pending	pendingpending.
Invariance to affine maps of f, end to end	Pending	True by construction for the listed probes; the end-to-end runs are not done.
An installable release on PyPI	Done	0.3.0 on PyPI (0.2.0 yanked). The code is public (https://github.com/BjBodner/amm-optimizer). The preregistrations and FT1 data are in a research archive whose DOI is pendingpending.

6. How many variants and cycles were tried

A rule set built on 24 functions invites the objection that it is a decision tree fitted to them. The design effort is therefore counted, not hidden.

• Cycles. 62 development cycles were logged when the paper draft was written, in 74 numbered batches, with 25 holdout or fresh-id confirmation runs and 21 promoted composites. The batch index has grown since: it reaches cycle 71 at the freeze. See the research log for the loop, the failed predictions and the freeze.
• Arms screened and CPU hours. pendingpending. These are to be reconstructed from the arm files and run logs.
• Spent instance ids. Every confirmation used a block of bbob instance ids that no earlier decision had used, and that block is then spent. At the freeze: spent 1-100 and 116-769, reserved and never used 101-115, next free 770. The ledger is a file (docs/research/batches/spent_ids.json) in the research archive (DOI: pendingpending, pending), not in the public code repository. The analysis code checks it before any claim is scored.
• Rejected and parked. The log keeps rejected candidates, including some that passed the screen and failed the confirmation. The research log tabulates them.

7. How the in-sample numbers differ from FT1

	Development field (in-sample, section 2)	Final test FT1 (section 1)
Instances	bbob ids 61-70	bbob ids 101-110, never used in any decision
Use of the data	compared with the field after each composite; steered about twenty composites	run once on the frozen configuration
Runs	one seed per instance, no interval	one run per cell and instance, seed 9000; the means over cells carry bootstrap intervals (F1, F4)
Cell rule	ahead, level or behind by 0.02 on the mean, a plotting threshold	credibly behind if some competitor leads by more than 0.02 with posterior probability above 0.9 over the paired runs; credibly ahead against every competitor that ran the cell
Configuration	a composite of c45 and c46 runs (section 8)	the frozen configuration itself
Competitors	uneven coverage (section 8)	the same field; HCMA at D 2 to 20, TPE on a capped slice
Behind cells	108in-sample of 600 under the 0.02 rule	35 of 600 under the credible rule

The two behind counts answer different questions, because the rules differ. Neither is a forecast of the other, and a difference between them is not a measure of how much the development numbers overstated the result. The comparison that tests the development result is the one against the preregistered ranges in section 1 and on the research log.

8. Caveats about the development data

1. The c46 series is a composite. 528 of the 600 cells come from full-grid runs of the previous configuration c45. Only 72 cells (D 10, 20 and 40 at 100 times the dimension) were run with the literal c46 configuration, which differs from c45 by one parameter from D = 10. The other cells are assumed identical and were not rerun. The frozen configuration’s hash is 2153948ebed51150; at the time of the in-sample runs it was 9f570d51aa12852a. The config text is the same; the hash changed because the configuration schema gained an opt-in field.
2. Uneven competitor coverage. HCMA has no D = 40 values. TPE ran a capped slice only. NGOpt has no D = 10 runs (the final test has NGOpt at every dimension). Where a competitor is missing, the best competitor is the best of the others, so those cells are easier to win than a seven-way comparison suggests. The list above names the missing competitors in each cell.
3. Run records. Most run files were produced in development worktrees with uncommitted changes, and the Optuna and Nevergrad versions and the hardware were not recorded. See the reproduce page.
4. Seed variance is not measured. One seed per instance and ten instances give no estimate of run-to-run spread: pendingpending.

9. A no-free-lunch note

No optimizer outperforms another when performance is averaged over all possible objective functions (Wolpert and Macready, 1997, “No free lunch theorems for optimization”). AMM’s results are therefore claims about a class of problems, not about optimization in general. It uses structure that bbob functions have and that a random function does not: a known budget, locally smooth or separable or multi-basin structure that cheap probes can detect, and exact evaluations. Each probe has a default outcome so that a misreading costs evaluations and not the run, but a misreading is still possible on a class unlike bbob’s, and the 24 functions it was developed on are also the 24 it was confirmed on, in new instances only. Whether the probes read mechanisms and not function identities is the question the problems beyond bbob are meant to test, and it is open: not yet testedpending.

Still pending on this page

• per-evaluation overhead: pendingpending;
• seed-to-seed variance: pendingpending;
• the ablation: pendingpending;
• results beyond bbob: not yet testedpending.