Research article · Pre-registration
Can a model sleep its way to grokking sooner?
Results are in: Results: can a model sleep its way to grokking sooner?
Abstract
Small transformers trained on modular addition memorize first and generalize ("grok") thousands of epochs later, using a handful of Fourier frequencies. We test whether periodic "sleep" phases, short bouts of self-distillation toward the part of the model's output its generalizing circuit produces, make grokking happen sooner than the best tuned weight decay. The test has 15 arms and 20 seeds paired by seed, including a hindsight positive control, a realistic online version, and six controls that would explain a speed-up without the circuit. We also test whether a score that needs no test labels can find the circuit's neurons before grokking. The decision rules are fixed here, before any full-run result exists.
1. Introduction
Interpretability can often name the circuit a network will eventually use. If we knew that circuit early, could we steer training toward it and skip the long memorization phase? Modular addition is the cleanest place to ask, because the generalizing circuit is known in closed form (Nanda et al., 2023) and grokking is slow and reliable (Power et al., 2022).
This is a gate, not a paper. A pass justifies the next stage: harder tasks and larger models. A fail should say what broke, which is why the test has a positive control that gets perfect information about the circuit. If even that doesn't help, the line stops.
2. Methods
Task and model. (a + b) mod 113, with a one-layer transformer (d = 128, 4 heads, d_mlp = 512, no LayerNorm), trained full-batch with AdamW (learning rate 10⁻³, betas 0.9 and 0.98) on 30% of all pairs. Weight decay is 1.0 unless an arm says otherwise. The data split and initialization depend only on the seed, so arms are paired by seed. Training stops at 95% test accuracy or at 20,000 epochs.
Sleep. From epoch 1,500, every 500 epochs, the model runs 100 full-batch steps of KL distillation on training inputs only. Because the inputs are enumerable, this rules out test leakage. The teacher is the model's own logits moved toward a target: softmax(logits + a·(target − logits)), where a is the strength.
Arms. 15 arms, 20 seeds each:
| Arm | What it tests |
|---|---|
| Weight decay 1, 2, 3, 5 | Tuned baseline. The best by median is "best weight decay" |
| Hindsight oracle (a = 0.3, 1) | Positive control. Target = logits projected onto the Fourier span of the key frequencies from that seed's finished run |
| Online oracle (a = 0.3, 1) | Same, with key frequencies detected during training from the embedding. The realistic version |
| Sham (a = 0.3, 1) | Same operation with 5 random non-key frequencies |
| Matched noise | Random logit change with the same per-example size as the oracle's |
| Anti | Removes the key-frequency components. Should slow grokking |
| Extra training | 100 extra normal training steps per sleep. Compute control |
| Shrink | Multiplies all weights by 0.95 per sleep. Uniform-downscale control |
| Soft logits | Teacher = half-scaled logits. Tests "sleep only shrinks the logits" |
Primary metric. Epochs to 95% test accuracy. Runs that never get there count as 20,001. Each arm is compared with best weight decay by a one-sided Mann-Whitney U test.
3. Decision rules
Gate A (30% data; needs at least 8 seeds in every arm):
- FAIL if the best hindsight arm is not faster than best weight decay (p ≥ 0.05). The sleep line stops.
- AMBIGUOUS if hindsight passes, but the sham at the same strength also beats best weight decay (p < 0.05), or matched noise, shrink, soft logits or extra training has a median at or below hindsight's.
- PASS otherwise. We then report whether the online version passes too, and whether "anti" slows grokking.
Gate B (lower priority) repeats the comparison at 20% and 25% data, where uniform methods struggle: does the oracle grok where weight decay doesn't, or much sooner? It is interpreted only if Gate A passes.
Gate C asks whether a score that needs no test labels can find the circuit early. Per-neuron gradient coherence, ‖mean_i g_ij‖² / mean_i ‖g_ij‖², is measured at epochs 1,500, 2,500, 4,000 and 6,000 in the weight-decay-1 runs. It is compared with labels from the grokked model: a neuron counts as a Fourier neuron if more than 80% of its activation power is on the key frequencies. PASS: mean AUROC ≥ 0.75 at some epoch ≤ 4,000. A fail rejects this score, not the idea.
4. Observations made while building (not verdicts)
On one seed during development, hindsight sleep reached 95% test accuracy at epoch 2,350, against 6,500 for weight decay 1.0. The gradient-coherence score was at chance on that seed. One seed is an anecdote; the run uses 20.
5. Protocol and amendments
- Rules are frozen in the project's
SPEC.md. Any change goes under a dated "Amendments" heading with the reason. Analyses not in the spec are reported as exploratory. - No seeds or arms are added to push a result over a threshold.
- Gate A may need a second night to reach 20 seeds in every arm. If so, the report says so, and gives no verdict below 8 seeds.
- Amendments so far, both before the full run and neither changing an arm, metric, sample size or pass rule: the code was moved from a combined bundle into its own project, and the worker count was set from a timing benchmark on the M1 (about 2.2 minutes per run).
- Amendment, November 2, 2026: the run started at 7:58 pm Mountain, not 10 pm as first planned. The launcher scheduled for 10 pm had stopped, so the run was started by hand. This article was published before the run started, and no arm, metric, sample size or pass rule changed.
6. Verifying this pre-registration
The code is not public yet. The frozen spec has SHA-256 hash 04e695c4358001a2cad5351948dcced6ef51585b2de0ebe52d156cbb627c4fac, committed November 2, 2026 at 11:08 Mountain in commit 5a51e5e. When the code is released, anyone can confirm the spec matches.
7. Limitations
- One toy task. Modular addition has a known, clean circuit. A pass says the mechanism can work where the circuit is known, not that it transfers.
- The positive control cheats on purpose. The hindsight oracle uses each seed's finished run. Only the online arm is a method someone could use.
- Laptop scale. All runs fit on one 16 GB M1 overnight.
References
- Nanda, N., Chan, L., Lieberum, T., Smith, J. & Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. ICLR.
- Power, A., Burda, Y., Edwards, H., Babuschkin, I. & Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177.