Research article · Pre-registration

Can a model sleep its way to grokking sooner?

Kāula research note, November 2, 2026. Pre-registration: the metrics, sample sizes and pass rules below were fixed before tonight's run. Results follow in a companion note.

Results are in: Results: can a model sleep its way to grokking sooner?

Abstract

Small transformers trained on modular addition memorize first and generalize ("grok") thousands of epochs later, using a handful of Fourier frequencies. We test whether periodic "sleep" phases, short bouts of self-distillation toward the part of the model's output its generalizing circuit produces, make grokking happen sooner than the best tuned weight decay. The test has 15 arms and 20 seeds paired by seed, including a hindsight positive control, a realistic online version, and six controls that would explain a speed-up without the circuit. We also test whether a score that needs no test labels can find the circuit's neurons before grokking. The decision rules are fixed here, before any full-run result exists.

1. Introduction

Interpretability can often name the circuit a network will eventually use. If we knew that circuit early, could we steer training toward it and skip the long memorization phase? Modular addition is the cleanest place to ask, because the generalizing circuit is known in closed form (Nanda et al., 2023) and grokking is slow and reliable (Power et al., 2022).

This is a gate, not a paper. A pass justifies the next stage: harder tasks and larger models. A fail should say what broke, which is why the test has a positive control that gets perfect information about the circuit. If even that doesn't help, the line stops.

2. Methods

Task and model. (a + b) mod 113, with a one-layer transformer (d = 128, 4 heads, d_mlp = 512, no LayerNorm), trained full-batch with AdamW (learning rate 10⁻³, betas 0.9 and 0.98) on 30% of all pairs. Weight decay is 1.0 unless an arm says otherwise. The data split and initialization depend only on the seed, so arms are paired by seed. Training stops at 95% test accuracy or at 20,000 epochs.

Sleep. From epoch 1,500, every 500 epochs, the model runs 100 full-batch steps of KL distillation on training inputs only. Because the inputs are enumerable, this rules out test leakage. The teacher is the model's own logits moved toward a target: softmax(logits + a·(target − logits)), where a is the strength.

Arms. 15 arms, 20 seeds each:

ArmWhat it tests
Weight decay 1, 2, 3, 5Tuned baseline. The best by median is "best weight decay"
Hindsight oracle (a = 0.3, 1)Positive control. Target = logits projected onto the Fourier span of the key frequencies from that seed's finished run
Online oracle (a = 0.3, 1)Same, with key frequencies detected during training from the embedding. The realistic version
Sham (a = 0.3, 1)Same operation with 5 random non-key frequencies
Matched noiseRandom logit change with the same per-example size as the oracle's
AntiRemoves the key-frequency components. Should slow grokking
Extra training100 extra normal training steps per sleep. Compute control
ShrinkMultiplies all weights by 0.95 per sleep. Uniform-downscale control
Soft logitsTeacher = half-scaled logits. Tests "sleep only shrinks the logits"

Primary metric. Epochs to 95% test accuracy. Runs that never get there count as 20,001. Each arm is compared with best weight decay by a one-sided Mann-Whitney U test.

3. Decision rules

Gate A (30% data; needs at least 8 seeds in every arm):

Gate B (lower priority) repeats the comparison at 20% and 25% data, where uniform methods struggle: does the oracle grok where weight decay doesn't, or much sooner? It is interpreted only if Gate A passes.

Gate C asks whether a score that needs no test labels can find the circuit early. Per-neuron gradient coherence, ‖mean_i g_ij‖² / mean_i ‖g_ij‖², is measured at epochs 1,500, 2,500, 4,000 and 6,000 in the weight-decay-1 runs. It is compared with labels from the grokked model: a neuron counts as a Fourier neuron if more than 80% of its activation power is on the key frequencies. PASS: mean AUROC ≥ 0.75 at some epoch ≤ 4,000. A fail rejects this score, not the idea.

4. Observations made while building (not verdicts)

On one seed during development, hindsight sleep reached 95% test accuracy at epoch 2,350, against 6,500 for weight decay 1.0. The gradient-coherence score was at chance on that seed. One seed is an anecdote; the run uses 20.

5. Protocol and amendments

6. Verifying this pre-registration

The code is not public yet. The frozen spec has SHA-256 hash 04e695c4358001a2cad5351948dcced6ef51585b2de0ebe52d156cbb627c4fac, committed November 2, 2026 at 11:08 Mountain in commit 5a51e5e. When the code is released, anyone can confirm the spec matches.

7. Limitations

References

All notes · The four pre-registrations