"Tiny" Recursive Reinforcement Learning

The Tiny Recursive Model (TRM) showed that surprisingly small networks can solve difficult reasoning problems by reusing the same parameters recursively.

Instead of making the model deeper, TRM repeatedly applies a small network to refine an internal representation and its prediction.

This led us to a simple question:

Can the same idea be useful for reinforcement learning?

We adapted TRM into a Q-network and compared it with standard MLP and CNN architectures.

The result was promising, but ultimately mixed: the recursive model sometimes learned faster early in training, but often converged to a worse solution than conventional architectures.

---

From TRM to a Q-Network

TRM maintains a prediction y together with a latent state z. A small network is then applied repeatedly:

z <- f(x, y, z)
y <- g(y, z)

The important part is that the parameters are shared between recursive steps.

Instead of adding more layers and parameters, we can therefore spend more computation by applying the same network multiple times.

To adapt this idea to reinforcement learning, we use the recursively refined representation to predict Q-values.

This introduces an interesting question that does not appear in ordinary supervised learning:

What should each recursive step be trained to predict?

In Q-learning, the target is itself bootstrapped from another value estimate:

Q_target = reward + gamma * max_a Q(next_state, a)

We experimented with three different ways of combining this target with TRM's deep supervision.

---

Three Ways to Supervise Recursion

TRM-Standard

The simplest approach is to compute one target using the final recursive depth:

T = reward + gamma * max_a Q_K(next_state, a)

and train every recursive prediction against that same target:

Q_1(state) -> T
Q_2(state) -> T
...
Q_K(state) -> T

So although the network produces predictions at multiple depths, the supervision signal remains fixed.

This is the closest analogue to standard deep supervision.

---

TRM-Iterated

In the second approach, each recursive depth gets its own target.

At depth k:

T_k = reward + gamma * max_a Q_k(next_state, a)

and we train:

Q_k(state) -> T_k

The target therefore changes as the recursive computation progresses.

In other words, prediction depth k is supervised using the target network evaluated at the same depth k.

---

TRM-Chain

The third version shifts the target by one recursive step.

Instead of training depth k from the target at depth k, we train it from the target produced at depth k-1:

T_(k-1) = reward + gamma * max_a Q_(k-1)(next_state, a)

Q_k(state) -> T_(k-1)

This version is interesting because the recursive computation starts to resemble repeated Bellman-style refinement.

Each step learns from the value estimate produced by the previous recursive step.

Of the three variants, this was the one where increasing recursive depth had the clearest effect.

---

A Small Test: Car-on-Hill

Before moving to larger environments, we tested the different approaches on Car-on-Hill.

This gave us a simpler setting where we could inspect how increasing the number of recursive supervision steps affected learning.

Standard

TRM generally performed well relative to the MLP baseline.

However, increasing the number of recursive supervision steps had surprisingly little effect. Models with 4, 8, or 16 recursive steps behaved quite similarly.

In this formulation, additional recursive computation did not clearly improve the learned value function.

---

Iterated

The Iterated formulation produced a very similar result.

Again, increasing recursive depth had only a small effect.

As a sanity check, we also trained a model with a single supervision step.

With only one step there is effectively no recursive refinement, so its behavior should be close to the MLP baseline. That is roughly what we observed.

---

Chain

The Chain formulation behaved very differently.

Here, increasing the number of recursive steps produced a clear change in learning behavior.

In particular, deeper variants performed substantially better than shallow ones.

Unlike Standard and Iterated supervision, the Chain formulation explicitly couples consecutive recursive predictions:

target at step 0 -> prediction at step 1
target at step 1 -> prediction at step 2
target at step 2 -> prediction at step 3
...

This makes increasing recursive depth more similar to increasing the number of Bellman-like refinement steps.

The Car-on-Hill results suggested that recursion can matter, but only if the training objective gives the recursive steps meaningfully different roles.

---

Scaling to MiniAtari

We next tested the recursive architecture on several MiniAtari environments.

We compared three architectures:

• Baseline-CNN — a conventional convolutional Q-network
• Baseline-MLP — an MLP operating on the environment observation
• RQN — our recursive Q-network based on TRM

In the plots below, RQN corresponds to the TRM architecture.

Alien	Breakout
DemonAttack	Enduro
Frostbite

The results were much less consistent than the Car-on-Hill experiments, even though there were some promising signs.

On Enduro, RQN learned much earlier than the CNN baseline and eventually reached the highest reward of the three architectures in these runs, even though the baseline CNN was still training. On Alien, RQN remained competitive and eventually achieved strong performance. These results were encouraging because they showed that the recursive architecture was capable of learning useful Q-functions in more complex environments.

But this behavior did not generalize. Frostbite illustrates the main failure mode particularly well. Early in training, RQN looked very promising. It improved much faster than the CNN and initially reached substantially higher rewards. But the CNN continued improving. RQN instead became more unstable and eventually converged to a much worse solution. On DemonAttack, RQN clearly outperformed the MLP baseline but remained far behind the CNN. On Breakout, the recursive model learned a reasonable policy but eventually plateaued below both conventional architectures.

This became the central result of the project:

Recursive Q-networks could improve early learning without reliably improving the final policy.

---

What Did Recursion Actually Buy Us?

The experiments suggest that recursion does affect reinforcement learning.

In several environments, RQN learned useful behavior unusually quickly. The Car-on-Hill experiments also showed that increasing recursive depth can have a strong effect when the supervision scheme explicitly connects consecutive recursive steps.

But these improvements were mostly visible in learning speed, not in the final solution.

A pattern we observed several times was:

Early training:
RQN > conventional network

Later training:
conventional network > RQN

---

Why Might This Happen?

One important difference between TRM's original setting and reinforcement learning is the nature of the target. In supervised tasks such as Sudoku, the correct output is fixed. Recursive computation can repeatedly refine a prediction toward that same solution. Q-learning is different. Its target is:

reward + gamma * max_a Q(next_state, a)

and that target is:

• bootstrapped from another prediction,
• non-stationary throughout training,
• and dependent on the value function being learned.

Recursive refinement therefore does not necessarily mean refinement toward a fixed correct answer. Instead, the model is repeatedly reasoning using estimates that are themselves changing. One possible explanation for our results is that recursive computation helps propagate useful value information quickly during early training.

But the same mechanism may also propagate approximation errors or encourage the network toward a suboptimal fixed point. That would be consistent with what we observed in environments such as Frostbite: fast initial learning followed by worse long-term convergence.

We do not have enough evidence to say that this is definitely the mechanism, but it seems like an interesting direction for further investigation.