Research article · Results
Results: can you score a learning rule without labels?
Verdict
Gate: PASS (pre-registered: rho >= 0.5 with CI lower bound > 0.3). Best label-free score: RankMe.
That's the analysis script's verdict line, exactly as written. RankMe passes. Across 108 rules, its Spearman correlation with linear-probe accuracy is 0.583, with a 95% CI of 0.443 to 0.695. LiDAR and cross-seed agreement don't pass.
As pre-registered, a pass picks the fitness score for the search: RankMe is the label-free score to evolve self-supervised local learning rules against.
1. Results
All 108 rules finished, with no errors: 8 known rules (including the untrained random-initialization network) and 100 random rules. Each label-free score is compared with linear-probe accuracy across the 108 rules.
| Label-free score | Spearman ρ vs probe accuracy | 95% CI | Rules |
|---|---|---|---|
| LiDAR | 0.341 | 0.162 to 0.498 | 108 |
| RankMe | 0.583 | 0.443 to 0.695 | 108 |
| Cross-seed kNN agreement, excess over random init | −0.242 | −0.412 to −0.055 | 108 |
| kNN accuracy (uses labels; reference only) | 0.930 | 0.900 to 0.952 | 108 |
The pass rule needs ρ ≥ 0.5 and a CI lower bound above 0.3. RankMe clears both. LiDAR's lower bound is above zero but its ρ is below 0.5. Cross-seed agreement is negatively correlated with probe accuracy: rules whose two seeds agree more tend to probe worse.
kNN accuracy uses labels, so it can't be a fitness score. It's reported as a check that the probe is measuring something stable: two label-based measures of the same representations agree closely (ρ = 0.930).
2. What else the report shows
These observations weren't part of the pre-registered rule, so they are exploratory:
- Most rules make the features worse than doing nothing. The untrained random-initialization network gets 0.375 probe accuracy (raw pixels get 0.356). Only 13 of the 107 learned rules beat it.
- The best rules are only modestly better. The top five by probe accuracy are cross_hebb_decorr (0.439), rand040 (0.437), rand098 (0.427), rand021 (0.424) and rand055 (0.424). Four of the five are random mixtures, not known rules.
- The analysis script suggests a next step. On a fail it would have printed "Try invariance-penalized variants first." That suggestion was never in the spec, and with a pass it doesn't apply.
3. What the verdict means
The search can use RankMe as its fitness score without labels. That's what the gate was for: an evolutionary search optimizes whatever it's given, and of the three scores tested, only RankMe ranks rules in roughly the order a linear probe does.
It doesn't mean RankMe can't be gamed. A correlation of 0.58 leaves a lot of room for a search to find rules that score well and probe badly, so the search should keep checking its best rules against a probe.
4. Deviations from the plan
- The run started at 7:58 pm Mountain on November 2, 2026, not 10 pm, as recorded in the pre-registration's amendments.
- At 8:32 pm the run was paused by hand, because other work had saturated the Mac. Four of the 108 rules had finished by then and were kept. The rule in progress had no saved result and was rerun from scratch.
- The run resumed at 9:32 pm, once the machine's load had dropped, and finished at 1:05 am on November 3 with all 108 rules done and no errors.
- No arm, metric, sample size or pass rule changed.
5. Limitations
- Most of the population is weak. Only 13 of 107 learned rules beat random initialization, so much of the correlation comes from telling bad rules from worse ones. This test doesn't say how well RankMe ranks rules among the good ones, which is where a search will spend its time.
- One network per score. Probe accuracy, LiDAR and RankMe are measured on the seed-0 network. The second seed is used only for cross-seed agreement, so each rule's own scores are noisy.
- Approximate CIs. The 95% intervals use the Fisher z-transform, not a bootstrap.
- Small networks, short training. A two-layer network trained for 1,500 steps on 20,000 CIFAR-10 images. A score that works here may not track quality in deeper networks or longer runs.
- A linear probe is one view of quality. We treat it as the reference, but it isn't the only one.
What's next
Evolve self-supervised local learning rules against RankMe, and check the best rules found against a linear probe as the search goes.