Neural networks need sleep
We borrowed neurons from the brain and left out sleep. In a toy test, a network that prunes and dreams every night keeps its old task at 96%. One that studies all night falls to 68%.
We borrowed neurons from the brain and left out the part where it sleeps. I think a model that learns day after day needs a night: a phase with no new input, where it recalls what it knows and is forced to say it with fewer weights.
The brain analogy was always loose. Nothing in the brain is known to do backpropagation. But sleep looks worth stealing: it replays the day’s memories, and is thought to consolidate them into long-term storage. It is how I learn too. Recall it, then sleep on it.
Doesn’t training already do this?
In a way, it does nothing else. Standard training shuffles everything the model will ever learn into every batch, so each step rehearses the old alongside the new. Every batch is a dream of old data. If anything, it is all sleep.
A night only earns its keep when a model learns the way we do: in order, one unshuffled day after another. Then today’s lesson overwrites yesterday’s.
What would a night look like?
My proposal is two moves, and no new data:
- Recall. The model makes up its own inputs and learns to answer them the way it did when it last woke up.
- Compress. It drops its weakest weights for good, so what it knows has to fit in what is left.
To try it, I trained two identical small networks. Both learn task A for two days, then task B for four. A day is 300 training steps on that day’s data only.
The insomniac spends each night studying the day’s data again. The sleeper gets no new input. It prunes the weakest 20% of its remaining weights, then dreams: it feeds itself random noise and matches the answers of the self that last woke up. No old data is stored. Between dreams it replays today’s examples, as the hippocampus is thought to.
- Insomniac68% accuracy100% of weights
- Sleeper96% accuracy26% of weights
- Day 1task A
- Day 2task A
- Day 3task B
- Day 4task B
- Day 5task B
- Day 6task B
Both start task B knowing task A at 98%. Four days later the insomniac remembers it at 68%. The sleeper remembers it at 96%, using 26% of its weights. Both reach 98% on task B. (Held-out data, mean of 8 runs.)
Watch the sleeper’s line. Every day its memory of task A sags, as low as 85% on the first. Every night it climbs back.
What is the catch?
It is a toy. The inputs have two dimensions, so random noise is a perfectly good dream: it covers everything the network could ever see. A real model would have to generate its own. A language model can literally talk in its sleep.
Dreams can also drift. The teacher is the model’s own previous self, so its mistakes can compound over many nights. That is the model-collapse worry.
And none of the parts is new. The wake-sleep algorithm (Hinton, Dayan, Frey and Neal) and complementary learning systems (McClelland, McNaughton and O’Reilly) both date from 1995. So does Robins’s pseudorehearsal, which is exactly dreaming on noise. Deep generative replay (Shin et al., 2017) and DeepInversion (Yin et al., 2020) generate the dreams instead. Pruning and distillation are the compression half.
What I am proposing is the routine: a night after every day, built into training.
We taught networks to study all day. The next thing to teach them may be bedtime.