Technical note

Can a language model choose its own layers, token by token?

Kāula research note, November 1, 2026; updated November 13, 2026 with Finding 7. Code, data and every number below: github.com/Kaula-AI/layer-looping.

A transformer runs every token through the same fixed stack of layers. We wanted to know whether that has to be so. Could each token take its own path through the network: loop a layer three times, skip the next, revisit an earlier one? Hard tokens would get more depth and easy tokens less.

We tested this on four models, from GPT-2 to a model trained to loop. Benchmarks were scored with paired statistics.

Summary

The idea

Take a model's layers as a library. The standard path for a token is 0, 1, 2, …, L−1. A per-token path might be

1 1 1 3 3 2 4 4 …   (layer 1 three times, skip 2, layer 3 twice, back to 2, layer 4 twice)

The cost of a path is the number of layer applications. The question is whether a cheap router can pick a good path for each token before the answer is known, at the same or lower compute.

How we measured it

Finding 1: the hindsight gain is large, and almost none of it is reachable

Hindsight vs router
Blue: loss reduction if each token gets its best single edit, chosen knowing the answer. Orange: the best trained router at zero extra compute. GPT-2 and Qwen3 are pilots that apply each edit to the whole sequence; Gemma-4 E2B and Ouro-1.4B use the exact per-token oracle.

Three checks say the hindsight gain is mostly selection, not signal (Finding 7 adds a direct control):

Every router we tried scored at or below chance at identifying which tokens an edit would help (AUC 0.20–0.52, where 0.5 is chance). That covered ridge and MLP routers, routers reading the current or the previous token, logit-lens and attention features or raw residuals, training-free entropy and norm rules from prior work, and "try a few edits and keep the most confident output." Every one preferred no edit at all when held to the base model's compute.

Finding 2: a standard model won't learn to use a loop from light fine-tuning

On Gemma-4 E2B we trained LoRA adapters with a looped block of layers 5–8. A learnable weight, starting at 0, mixed in the loop's output; we also tried a learned per-token gate. Training kept the mixing weight at |α| < 5×10⁻⁴, even with a 50× faster learning rate on it. The gate closed to 0.2%. Every looped variant landed within seed-to-seed noise of a LoRA control with no loop. Rerunning pretrained layers on their own output is out of distribution, and about 6M tokens of training didn't change that.

Finding 3: a router trained with the model works

Ouro-1.4B (ByteDance) runs its 24 layers four times and was pretrained on 7.7T tokens with a per-token exit gate. Using that gate to stop each token early:

Gate vs router
Blue: Ouro's own gate (threshold 0.2) minus a fixed 3 rounds for every token. Orange: our trained per-layer router (Ouro's weights frozen) minus Ouro's gate. Numbers in parentheses are layers per answer token. Paired 95% CIs over 1,000 items.

On held-out text, the gate at threshold 0.5 exits 54% of tokens early, saving 13 layers per token at a loss change of −0.002. Our frozen-model routers on the same model were worse than no edit (+0.057).

Finding 4: finer control lost to coarse control

We then trained our own router with the model's loss. At every one of Ouro's 96 unrolled layers, a small head chooses skip, once or twice for each token. It was trained end to end with straight-through Gumbel-softmax and a compute budget of 72 layers per token.

Finding 5: why our router lost

We ran the retrained router (held-out loss 2.417, matching the first run) against Ouro's gate on 32 held-out FineWeb-Edu chunks of 1,024 tokens.

Compute curve
Held-out loss against layers per token. Blue: Ouro's gate at thresholds 0.05 to 0.9. Green: every token runs the same number of rounds. Orange: our router and three controls.
Skip map
Share of tokens skipping each of Ouro's 96 unrolled layers.

The 4th round is worth only 0.025 nats per token on average, so dropping it whole is the cheapest compute to give up, and that is what Ouro's gate does. Our router skips 0% of round 4 and instead removes the middle of round 3, which costs more. It also never chose "run twice". Our reading, not yet tested directly: a head deciding one layer at a time sees that skipping any single late layer hurts, and can't see that skipping all 24 of them together is nearly free. Independent per-layer decisions trained with straight-through gradients have no way to discover coordinated, multi-layer actions like "stop here."

Finding 6: trained properly, a router recovers the gate, and extra loops still don't help

The diagnostic pointed to the training method, so we rebuilt the router around it:

On held-out FineWeb-Edu, all three land on Ouro's gate curve to within 0.001 nats (at 70 layers per token: 2.3372 vs the gate's 2.3362). The per-layer router was about 0.09 nats worse at the same compute, so the training method was the problem. But the router does not beat the gate. Retraining for 12M tokens recovers what Ouro learned over 7.7T, and the stop-only control comes out identical.

The router drove the "re-run half a round" option from 0.25% down to 0.01% of tokens. A direct check shows why:

Ouro only benefits from the loop it was trained on, whole rounds, and even a whole fifth round added nothing on the benchmarks.

Finding 7: random noise matches the hindsight gain, and coarser routing doesn't change that

Finding 1 argued that the hindsight gain is mostly selection. We tested that directly, with a rule fixed before running (E13). For every token we replaced each layer edit with a random nudge to the token's input to one layer, the size of that layer's own update, and took the best of the same number of nudges in hindsight.

Noise null
Left: hindsight gain against the number of options per token, for real layer edits (solid) and random nudges (dashed). Right: the same when a window of consecutive tokens must share one choice. 4,088 held-out positions per model.

Library of Layers, published the same week, routes windows of tokens rather than single tokens and reports that per-token routing was unstable to train. So we asked whether a frozen model's hindsight gain survives when neighbouring tokens must share one choice (E13b). It doesn't. The gain falls about 3× by windows of 8 tokens and 10× by 32, for real edits and noise alike, and is gone at one choice per 512-token chunk. E2B keeps a small edge over noise at 8–32 tokens (+0.02 to +0.04 nats). That's an upper bound: the screen adds up single-token effects and ignores how edits inside a window interact. Ouro keeps none.

What we take from this

  1. Per-token compute allocation is real, but it has to be learned jointly with the model. Fitting a router to a frozen network finds nothing; a gate trained alongside the network finds benchmark-level gains.
  2. The useful granularity so far is coarse. "How many rounds" beat "which layer, how many times," even though the second strictly contains the first. More options made the hindsight gain bigger and the learned policy worse: the per-layer router never discovered the coordinated "stop after round 3" that the gate gets for free. Coarseness alone isn't the fix, though: routing windows of tokens in a frozen model finds nothing either (Finding 7).
  3. Hindsight oracles mislead here. A per-token oracle over many edits measures how often some perturbation happens to favor the right token: random nudges of the same size score 93–102% as well. Report it next to a noise control, and don't read it as reachable headroom.
  4. A router can only spend computation the model already knows how to use. It can choose how much of the model's trained recurrence each token gets, and Ouro's gate already does that near-optimally. New kinds of looping have to be trained into the model itself.

Open questions

Limitations

Reproduce

All results are in results/; the experiment log is EXPERIMENTS.md.

python -m pytest -q tests                                               # 36 CPU exactness tests
python oracle.py --model ByteDance/Ouro-1.4B --n_tokens 20000 --out ouro_quick && python probe.py --path ouro_quick/oracle.npz --out_dir ouro_quick
python bench.py --model ByteDance/Ouro-1.4B --policies base rounds:3 gate:0.2 --out bench   # benchmarks
python routed_ouro.py --variant router --no_lora --target_layers 72 --out r5/router72_frozen  # our router
python bench_compare.py --dir bench ... --a gate:0.2 --b rounds:3 --metric acc_norm        # paired tests
python diagnose_router.py --ckpt r5/router72_frozen/model.pt --out diag                     # why the router lost
python structured_router.py train --out sr && python structured_router.py eval --heads sr/heads.pt --out sr  # Finding 6
bash run_experiments.sh noise && MODEL=ByteDance/Ouro-1.4B bash run_experiments.sh noise   # Finding 7: noise control
python window_selection.py --oracle results/gemma-4-e2b-e13/oracle.npz --noise results/gemma-4-e2b-e13/noise.npz --out ws  # window screen
python scripts/make_findings_figs.py                                      # these figures

All notes · Code and data