Research article · Results
Results: where does backprop clearly win?
Verdict
No architecture qualifies. On none of the four networks does backprop beat the better of feedback alignment and sign-symmetry by 10 points. The largest gap is 5.3 points, on the 3-layer convnet.
As pre-registered, there is no headroom at this budget, so rule discovery waits for deeper networks and longer training on rented GPUs.
1. Results
CIFAR-10 test accuracy, mean of 2 seeds, after 4 epochs on 25,000 training images.
| Network | Backprop | Feedback alignment | Sign-symmetry | Gap (backprop − best alternative) |
|---|---|---|---|---|
| MLP, 3 hidden layers | 48.9% | 25.7% | 48.7% | 0.2 points |
| Convnet, 3 layers | 54.7% | 49.4% | 46.7% | 5.3 points |
| Convnet, 5 layers | 65.8% | 55.0% | 64.6% | 1.2 points |
| Convnet, 7 layers | 69.0% | 48.9% | 66.5% | 2.5 points |
2. What the numbers say
The verdict turns on sign-symmetry. Feedback alignment does fall well behind backprop: by 23 points on the MLP, 11 on the 5-layer convnet and 20 on the 7-layer convnet. But the rule compares backprop with the better alternative, and sign-symmetry stays within 2.5 points of backprop everywhere except the 3-layer convnet.
The following observations weren't part of the pre-registered rule, so they are exploratory:
- Feedback alignment gets worse with depth. It falls from 55.0% at 5 layers to 48.9% at 7, while backprop and sign-symmetry keep improving. That's the pattern expected when random feedback can't align through many layers.
- The one sizeable gap is the least reliable. On the 3-layer convnet, sign-symmetry scored 55.0% on one seed and 38.4% on the other, so the 5.3-point gap there rests on one unstable run.
- Four epochs is short. Rankings could change with longer training. The pre-registered budget was chosen to fit a laptop, not to settle this question.
3. Deviations from the plan
- The run started at 7:58 pm Mountain, not 10 pm, as recorded in the pre-registration's amendments. All 24 jobs finished by 8:20 pm, with no errors.
- Other work was competing for the Mac and its GPU during the run. That slowed it down but doesn't change the accuracies. The run was paused at 8:32 pm, after this gate had finished.
- No other changes.
4. Limitations
- Two seeds. Enough to rule out a 10-point gap here, not to measure small differences.
- Short schedule. 4 epochs on half of CIFAR-10.
- Two alternatives. Other backprop-free rules weren't tested.
What's next
The rule-discovery search needs a testbed where every known alternative to backprop clearly loses. That means deeper networks and longer schedules than fit on one laptop overnight. That step moves to rented GPUs.