Sampling Luck Masquerades as Allocation Gain: Auditing Test-Time Budget Allocation for Neural Combinatorial Optimization
Why AI solvers' improvements often vanish when properly tested
Researchers discovered that neural optimization solvers report improvements from smarter budget allocation that simply don't exist — they're statistical mirages created by testing on the same data used to find the allocation. When tested fairly on held-out data, the 2–3% gains vanish completely. However, under real-world conditions where data shifts, adaptive allocation does deliver genuine 11–12% gains, but only for some solvers.
This finding catches a widespread testing flaw that makes optimization algorithms look better than they are. When companies or researchers evaluate AI solvers this way, they publish fake improvements and waste effort optimizing something that doesn't work. The authors provide a correction procedure and checklist so future evaluations don't repeat this error — and show where real gains actually hide.