The Matthew Effect in RL for LLMs: Why Early Success Can Sabotage Learning
Learning to solve hard problems in RL for LLMs by never giving up
In this blog post, Michael Noukhovitch introduces the Matthew Effect in reinforcement learning post-training for LLMs: early successes bias the model toward easy problems, causing it to neglect harder ones. He proposes 'Never Give Up' (NGU), a method that forces the model to keep trying on difficult problems. Experiments on Olmo 3.1 RL-Zero Math and code tasks show NGU improves performance, though limitations remain.
Training Olmo 3 7B base with RL on Dolci RL-Zero math improves its overall math ability. Or does it?
- austin-cheney
I have something like this very recently.
About a month ago I wrote a perf tool for my application to measure its network message speed. It showed I could send 3,250,000 WebSocket messages per second in my TypeScript application. If I moved each test of an experiment into its own call stack I could send 3,250,000 messages as fast as 0.002 seconds. Claude confirmed the number.
It was about two weeks later I learned the number is wrong. The number would be correct if this were just a linear instruction processing problem. Actually the processing time, CPU and network handling, is about 0.08%. The other 99.92% of processing time is in memory. What was happening is the perf tool would push messages into a queue on the socket from a JavaScript loop, but the speed to process these messages and drain the network buffer was less than the loops execution speed.
Correcting for that dropped my transmission speed to a more realistic 100,000 messages per second. Modified queue management then allowed a rate of 800,000 messages per second. Pushing each message send into a separate call stack then allowed for a rate of 6,000,000 messages per second on a single socket but it sacrifices stability to reach that rate of transmission speed.
My wrong number was confirmed by Claude for the same reason I believed it, a confirmation bias that was well reasoned from the logic. Claude came to confirm the number for the same reason I did but came to that conclusion a completely different way.
- MiliasGeiger
> For a deeper, more technical dive, check out the paper on arxiv and code on github.
These links at the top point to the same page, not the paper or the code!
- tpoacher
Why are two good quality comments on this thread dead on arrival?
I've seen this happen a lot recently, when the page still has relatively few comments. Insightful comments are dead leaving behind trivial ones.
What's going on?