Skip to content
mutome

How Mutome tests an idea.

A proposal starts an experiment. It is not evidence. Mutome bounds each candidate, measures it under a fixed protocol, reviews the results before publication, and measures its own progress on held-out labs.

  1. Propose a bounded hypothesis

    The model proposes a hypothesis and a candidate within an approved domain. Each campaign fixes its data, editable object, evaluator and budgets before the first call.

  2. Measure and attempt to reproduce

    Code applies the frozen protocol, compares the candidate with a reference and retains failures. The final test stays separate from the search. A simulation tests a model; it is not a new physical observation.

    A completed execution does not validate the hypothesis. A timeout remains inconclusive. Multiple replays alone do not demonstrate statistical significance.

  3. Review before publishing

    A person approves an exact version with its results, reproduction, limitations and usage rights. A scientific change requires new approval. Corrections and retractions retain their history.

  4. Measure the lab itself

    Mutome’s progress is measured on Slean labs it has never seen. In each one, the answers are hidden and every experiment is metered.

    Lift is the score of the current harness version minus the score of the raw model, at equal budget. A change to Mutome’s own harness is promoted only when its lift on these held-out labs is positive.

    The first battery was saturated: lessons from earlier labs changed the results by 0 and by +1 discovery. On harder, “compressible” worlds, the same lessons lifted results on three held-out seeds: 13, 13 and 17 verified discoveries, against 4, 6 and 5 for the raw agent. This is the first harness change that improved results on worlds Mutome had never seen. With one run per arm and seed, it will be repeated before lessons become the default.

Where we are.

Mutome is a research lab under construction. Here is what runs, what is still missing, and what comes next. The detailed measurements are on the Slean page.

See the measurements

What runs today

  • Bounded campaigns with frozen evaluators, budgets, and leases
  • Pause, cancel, and a global stop
  • A cost ledger capped at $50 a month
  • Replay, artifact hashing, backup and restore
  • Hash-bound approval and retraction for publication
  • Campaigns so far: synthetic oscillator tasks and a code-change autotune loop; most attempts were discarded or inconclusive
  • Slean labs, where Mutome measures itself

Not yet

  • Continuous, autonomous operation
  • Self-improvement confirmed by repeated measurements
  • Any discovery in real-world science
  • A result approved for publication: the list of published works is empty

What comes next

  • Repeat the compressible-worlds measurement, then make lessons Mutome’s default memory for Slean labs if the lift holds
  • Run Slean labs continuously within a $50 monthly budget
  • Change Mutome’s own harness only when it lifts results on held-out labs

Inspirations.

  • Karpathy’s autoresearch loop

    An agent proposes a change, a fixed evaluation measures it, and the change is kept or reverted.

  • Evolutionary search

    Many variants, selected by a fixed measure rather than by the confidence of whoever proposed them.

  • Lean and Mathlib

    A small trusted kernel checks every proof, and a shared library lets verified results build on one another.

  • Dream-RSI

    The idea of replaying histories of discovery to learn how discoveries are made.