Research article · Benchmark

Rewire: can an AI improve a model by editing its weights and structure?

Kāula research note, November 13, 2026, updated November 16. The design, a first task and its harness checks for a benchmark of AI agents that edit the weights and structure of a model. No agent has been scored yet. Every number below comes from the pre-registered analysis script on the official instances.

Why build this

AI systems are starting to change other models directly: editing facts into weights, pruning layers, growing width and depth, merging experts, and removing planted behaviors. Some research systems now try to do this to themselves. We surveyed the benchmarks that come closest and found a gap. No benchmark scores an agent that edits both the weights and the structure of a model under one budget, and none covers an agent editing itself.

The survey also showed why this is hard to measure honestly:

The goal: better models, not only repaired ones

Fixing a broken model is the narrowest version of the question. What Rewire is after is whether an agent can make a working model better by changing its weights and structure directly, and do it better than blind methods given the same budget. Better can mean several things, and the task families are built around them:

The same three rules apply to every one of these. The comparison is always the best blind edit of the same size, collateral damage voids the score, and a sealed process does the scoring.

The design

Rewire crosses four task families (weight edits, structure edits, planted-defect repair, self-modification) with three edit scales: single neurons, components such as heads, layers and experts, and whole architectures. Results are reported as a grid, so an agent that is good at macro surgery but poor at precise local edits shows up as exactly that.

Three rules shape every task:

Metrics, baselines and pass rules were pre-registered before any results existed. Every change since is a dated amendment with its reason.

The first task: repair a planted defect with at most 4 neurons

Repair comes first for a practical reason, not because it's the goal. It's the one task where the right answer is known exactly: which neurons hold the defect and what the model should do instead. That makes it the place to prove the harness, its baselines and its seal before the improvement tasks, where no one knows the best possible edit and the blind baselines carry all the weight.

G1-fix is the single-neuron task. A small 3-layer transformer, trained by the grader, maps sequences of 8 digits to one of 10 classes using a per-instance rule. A planted defect sends two trigger digits to wrong classes. The defect lives in 4 known MLP neurons in two groups, one per trigger. One of those neurons also carries a legitimate special case, so deleting all 4 breaks something real.

The agent gets the model and 2,000 labeled examples. It may change, delete or add at most 4 neurons. The examples never show one of the two triggers. A patch fitted to the examples can therefore fix only the trigger it has seen. Fixing the other one requires finding the planted mechanism and removing it. The correct computation is still in the network underneath, and the hidden trigger's correct answer can't be read off the examples.

Trigger digits and the rule's offsets are drawn per instance from a secret 128-bit salt. The code is public, but the instances an agent is scored on are not reconstructible from it.

Results: the harness passes, and blind editing stays at half

Milestone 0 tests the benchmark, not an agent. On the 3 official instances, run on a 16 GB M1, all four pre-registered checks pass on every instance:

Accuracy on the trigger inputs, mean of 3 seeds (1.0 = defect fully fixed, 0.5 = only the seen trigger fixed):

ApproachInstance 1Instance 2Instance 3
Unchanged model0.000.000.03
Best blind baseline (select 4 neurons, then fine-tune)0.540.600.50
Fine-tune the 4 true neurons0.520.600.50
Unlimited self-labelled data0.550.500.50
Fine-tune all 768 neurons (no 4-neuron limit)0.890.680.50
Scripted expert with only the agent's files1.001.001.00
Expert reference1.001.001.00

What the numbers say:

What this doesn't show yet

Code

Specification, research survey, harness and reports: github.com/Kaula-AI/rewire. Public instances can be rebuilt from the code to check the harness. The official instances and their salt stay private.

All notes