Benchmark.
Compare a candidate with a reference under rules fixed in advance.
The evaluator, the reference, the budget, and the held-out data are frozen before a candidate is judged. The answer is one of three: better, not better, or inconclusive.
Runs today on synthetic oscillator protocols and Slean labs.
See an example
Fix the comparison before seeing the result.
A candidate, a reference, an evaluator, and held-out data meet under one protocol and one budget. Improvements, failures, and uncertainty stay distinct.
Did the new idea actually beat the old one?
The problem
It is easy to make a candidate look better: tune it on the test, give it more budget, or forget the runs that failed.
Why it is hard
Small differences often sit within noise, and a run that times out says nothing either way. A fair comparison has to be designed before anyone sees the result.
How Mutome approaches it
Benchmark freezes the evaluator, the reference, the budget, and the held-out data first, gives both sides the same budget, and keeps inconclusive cases in the record instead of rounding them up.
Candidate versus reference, one frozen test
A synthetic oscillator protocol, step by step. The bars are an illustration of the procedure, not a measurement.
Two batteries today
Synthetic oscillator protocols (rediscovering a model, telling two models apart), and Slean labs: the held-out battery Mutome uses to measure itself, where an agent is scored against hidden answers and compared with a scripted reference at equal budget.
The evidence boundary
Both batteries are synthetic. They measure behaviour on defined tasks; they are neither a general score for the lab nor evidence about real-world science.