Research article · Results

Results: can a model sleep its way to grokking sooner?

Kāula research note, November 3, 2026. Results for the pre-registered gate in Can a model sleep its way to grokking sooner?. Every number below comes from the pre-registered analysis script and its raw results.

Verdict

Gate A: FAIL (positive control). Even the hindsight oracle (oracle_hindsight@1) does not beat wd5. Stop: guided sleep doesn't work with a perfect target.

Gate C: FAIL (pre-registered: AUROC@0.8 >= 0.75 at some epoch <= 4000). A FAIL here means this particular score doesn't work, not the idea; try attribution-based scores next.

Those are the analysis script's verdict lines, exactly as written. Gate A had all 20 seeds in every arm, well above the 8 the rule needs. The sleep line stops here as pre-registered. Gate B is not interpreted, because the spec only interprets it after Gate A passes.

One caveat belongs next to the verdict, though it doesn't change it: as the experiment was designed, no sleep arm could have beaten weight decay 5. Section 3 explains why.

1. Gate A: 30% of the data, 15 arms, 20 seeds

All 300 runs finished and every one reached 95% test accuracy, so no run was censored. Best weight decay by median was 5.0. The p-value is a one-sided Mann-Whitney U test of each arm being faster than weight decay 5.

ArmSeedsGrokkedMedian epochs to 95%95% CIp vs best weight decay
Weight decay 5 (best)2020800750–950—
Weight decay 320201,4751,300–1,9251.000
Hindsight oracle, a = 120202,3002,250–2,4251.000
Online oracle, a = 120202,3002,200–2,4001.000
Weight decay 220202,8002,225–3,5001.000
Sham, a = 120203,6753,425–3,8001.000
Hindsight oracle, a = 0.320204,8504,275–5,2751.000
Online oracle, a = 0.320204,8754,300–5,2751.000
Anti, a = 0.320204,9504,350–5,3751.000
Sham, a = 0.320204,9504,350–5,5001.000
Matched noise, a = 0.320205,0504,350–5,4501.000
Soft logits20205,2004,350–5,5001.000
Shrink20205,4504,825–5,8751.000
Extra training20205,5004,725–6,0001.000
Weight decay 120206,3505,425–6,9251.000

The positive control fails. The best hindsight arm (a = 1) has a median of 2,300 epochs, nearly three times weight decay 5's 800 (p = 1.000). Under the rule, that is a FAIL, and the AMBIGUOUS and PASS branches are never reached.

The online oracle doesn't pass either. Its a = 1 arm also has a median of 2,300 (p = 1.000).

"Anti" didn't slow grokking. The pre-registration asks about this only after a pass, so this is descriptive. All sleep arms train at weight decay 1, and anti's median of 4,950 is below weight decay 1's 6,350. It's about the same as the sham at the same strength (4,950) and the hindsight oracle (4,850).

2. Gates B and C

Gate B did not complete. It was the lower-priority queue, and the run reached its time limit with 54 of its 280 runs finished and 5 more skipped, leaving 221 never started. At most 5 seeds per arm finished, against 20 planned. The skipped runs were hindsight-oracle runs whose weight-decay-1 reference never grokked, so there were no key frequencies to project onto: four at 20% data and one at 25%. The report's Gate B tables are in the raw output, but Gate A failed and the seeds are too few, so we don't interpret them.

Gate C fails. Gradient coherence doesn't find the Fourier neurons. It was measured in the 23 weight-decay-1 runs that grokked: the 20 Gate A runs at 30% data, plus 3 Gate B runs at 25% data that finished. The analysis script pools both.

EpochMean AUROC (Fourier label > 0.5)Mean AUROC (> 0.8, pre-registered)Mean Spearman vs key-frequency fraction
1,5000.4890.5070.001
2,5000.5020.5310.040
4,0000.4900.5230.025
6,0000.4990.5810.104

The pass bar was a mean AUROC of 0.75 by epoch 4,000. The best score by then is 0.531, close to chance (0.5). As pre-registered, this rejects this particular score, not the idea of finding the circuit early.

3. Why no sleep arm could have passed (exploratory)

This is analysis after the fact, not part of the pre-registered rule, and it doesn't change the verdict.

All sleep arms train at weight decay 1, and sleep only begins at epoch 1,500. Seeds share the data split and initialization, so until epoch 1,500 every sleep run is exactly the matching weight-decay-1 run. Weight decay 5 turned out to be very fast: 18 of its 20 runs reached 95% before epoch 1,500, before any sleep arm had slept once. In the raw results, the earliest any sleep-arm run reached 95% was epoch 2,000. A sleep arm's median could not come near 800 epochs.

So the gate compared sleep on top of the slowest baseline with the fastest one. The weight-decay grid was meant to make the baseline strong, and it did, but the sleep arms didn't get the same tuning.

Within weight decay 1, the results do show something:

These comparisons are not pre-registered, so they suggest an experiment rather than make a claim.

4. Deviations from the plan

5. Limitations

What's next

As pre-registered, the sleep line stops: the positive control didn't beat best weight decay. Any follow-up, such as sleep on top of the best weight decay, or sleep starting before the baseline groks, would need its own new pre-registration. Gate C's suggested next step, attribution-based scores, was printed by the analysis script and isn't part of the spec.

All notes · The four pre-registrations