Research article · Results
Results: can a model sleep its way to grokking sooner?
Verdict
Gate A: FAIL (positive control). Even the hindsight oracle (oracle_hindsight@1) does not beat wd5. Stop: guided sleep doesn't work with a perfect target.
Gate C: FAIL (pre-registered: AUROC@0.8 >= 0.75 at some epoch <= 4000). A FAIL here means this particular score doesn't work, not the idea; try attribution-based scores next.
Those are the analysis script's verdict lines, exactly as written. Gate A had all 20 seeds in every arm, well above the 8 the rule needs. The sleep line stops here as pre-registered. Gate B is not interpreted, because the spec only interprets it after Gate A passes.
One caveat belongs next to the verdict, though it doesn't change it: as the experiment was designed, no sleep arm could have beaten weight decay 5. Section 3 explains why.
1. Gate A: 30% of the data, 15 arms, 20 seeds
All 300 runs finished and every one reached 95% test accuracy, so no run was censored. Best weight decay by median was 5.0. The p-value is a one-sided Mann-Whitney U test of each arm being faster than weight decay 5.
| Arm | Seeds | Grokked | Median epochs to 95% | 95% CI | p vs best weight decay |
|---|---|---|---|---|---|
| Weight decay 5 (best) | 20 | 20 | 800 | 750–950 | — |
| Weight decay 3 | 20 | 20 | 1,475 | 1,300–1,925 | 1.000 |
| Hindsight oracle, a = 1 | 20 | 20 | 2,300 | 2,250–2,425 | 1.000 |
| Online oracle, a = 1 | 20 | 20 | 2,300 | 2,200–2,400 | 1.000 |
| Weight decay 2 | 20 | 20 | 2,800 | 2,225–3,500 | 1.000 |
| Sham, a = 1 | 20 | 20 | 3,675 | 3,425–3,800 | 1.000 |
| Hindsight oracle, a = 0.3 | 20 | 20 | 4,850 | 4,275–5,275 | 1.000 |
| Online oracle, a = 0.3 | 20 | 20 | 4,875 | 4,300–5,275 | 1.000 |
| Anti, a = 0.3 | 20 | 20 | 4,950 | 4,350–5,375 | 1.000 |
| Sham, a = 0.3 | 20 | 20 | 4,950 | 4,350–5,500 | 1.000 |
| Matched noise, a = 0.3 | 20 | 20 | 5,050 | 4,350–5,450 | 1.000 |
| Soft logits | 20 | 20 | 5,200 | 4,350–5,500 | 1.000 |
| Shrink | 20 | 20 | 5,450 | 4,825–5,875 | 1.000 |
| Extra training | 20 | 20 | 5,500 | 4,725–6,000 | 1.000 |
| Weight decay 1 | 20 | 20 | 6,350 | 5,425–6,925 | 1.000 |
The positive control fails. The best hindsight arm (a = 1) has a median of 2,300 epochs, nearly three times weight decay 5's 800 (p = 1.000). Under the rule, that is a FAIL, and the AMBIGUOUS and PASS branches are never reached.
The online oracle doesn't pass either. Its a = 1 arm also has a median of 2,300 (p = 1.000).
"Anti" didn't slow grokking. The pre-registration asks about this only after a pass, so this is descriptive. All sleep arms train at weight decay 1, and anti's median of 4,950 is below weight decay 1's 6,350. It's about the same as the sham at the same strength (4,950) and the hindsight oracle (4,850).
2. Gates B and C
Gate B did not complete. It was the lower-priority queue, and the run reached its time limit with 54 of its 280 runs finished and 5 more skipped, leaving 221 never started. At most 5 seeds per arm finished, against 20 planned. The skipped runs were hindsight-oracle runs whose weight-decay-1 reference never grokked, so there were no key frequencies to project onto: four at 20% data and one at 25%. The report's Gate B tables are in the raw output, but Gate A failed and the seeds are too few, so we don't interpret them.
Gate C fails. Gradient coherence doesn't find the Fourier neurons. It was measured in the 23 weight-decay-1 runs that grokked: the 20 Gate A runs at 30% data, plus 3 Gate B runs at 25% data that finished. The analysis script pools both.
| Epoch | Mean AUROC (Fourier label > 0.5) | Mean AUROC (> 0.8, pre-registered) | Mean Spearman vs key-frequency fraction |
|---|---|---|---|
| 1,500 | 0.489 | 0.507 | 0.001 |
| 2,500 | 0.502 | 0.531 | 0.040 |
| 4,000 | 0.490 | 0.523 | 0.025 |
| 6,000 | 0.499 | 0.581 | 0.104 |
The pass bar was a mean AUROC of 0.75 by epoch 4,000. The best score by then is 0.531, close to chance (0.5). As pre-registered, this rejects this particular score, not the idea of finding the circuit early.
3. Why no sleep arm could have passed (exploratory)
This is analysis after the fact, not part of the pre-registered rule, and it doesn't change the verdict.
All sleep arms train at weight decay 1, and sleep only begins at epoch 1,500. Seeds share the data split and initialization, so until epoch 1,500 every sleep run is exactly the matching weight-decay-1 run. Weight decay 5 turned out to be very fast: 18 of its 20 runs reached 95% before epoch 1,500, before any sleep arm had slept once. In the raw results, the earliest any sleep-arm run reached 95% was epoch 2,000. A sleep arm's median could not come near 800 epochs.
So the gate compared sleep on top of the slowest baseline with the fastest one. The weight-decay grid was meant to make the baseline strong, and it did, but the sleep arms didn't get the same tuning.
Within weight decay 1, the results do show something:
- Every sleep arm groks sooner than weight decay 1 alone. Their medians run from 2,300 to 5,500 epochs, against weight decay 1's 6,350.
- At a = 1, the circuit target beats the sham. Hindsight's median is 2,300 (CI 2,250–2,425) and the sham's is 3,675 (CI 3,425–3,800). The intervals don't overlap.
- At a = 0.3, the target doesn't matter. Hindsight, online, anti, sham and matched noise all land between 4,850 and 5,050, so at that strength the effect looks generic.
These comparisons are not pre-registered, so they suggest an experiment rather than make a claim.
4. Deviations from the plan
- The run started at 7:58 pm Mountain on November 2, 2026, not 10 pm, as recorded in the pre-registration's amendments.
- At 8:32 pm the run was paused by hand, because other work had saturated the Mac (load average roughly 100 to 230, heavy swap). Six Gate A runs had finished by then and were kept. The three runs in progress had no saved result and were rerun from scratch.
- The run resumed at 9:32 pm, once the machine's load had dropped, and stopped launching new runs at its 8-hour limit. The last run finished at 5:34 am on November 3. 359 runs ended with a saved result (354 finished, 5 skipped), with no errors, and 221 Gate B runs never started.
- The pause and the load before it slowed the wall clock, but the metric is in epochs, so it isn't affected.
- No arm, metric, sample size or pass rule changed. The frozen spec still hashes to
04e695c4358001a2cad5351948dcced6ef51585b2de0ebe52d156cbb627c4fac, as published in the pre-registration. - Anything above that isn't in the spec is labelled exploratory.
5. Limitations
- The design flaw in section 3. The verdict is a fair reading of the pre-registered rule, but the rule couldn't test sleep against an equally tuned baseline. It shows that sleep at weight decay 1 doesn't beat weight decay 5, not that guided sleep can never help.
- One toy task. Modular addition with 113 residues and a one-layer transformer.
- Gate B is incomplete. At most 5 of 20 seeds per arm, so we can't say anything about lower data fractions.
- Gate C pools two data fractions. 3 of its 23 runs come from 25% data, not 30%. With AUROC near 0.5 at every epoch, this doesn't change the result.
- Laptop scale. All runs fit on one 16 GB M1.
What's next
As pre-registered, the sleep line stops: the positive control didn't beat best weight decay. Any follow-up, such as sleep on top of the best weight decay, or sleep starting before the baseline groks, would need its own new pre-registration. Gate C's suggested next step, attribution-based scores, was printed by the analysis script and isn't part of the spec.