Overview
Problem. Long-range estimates may depend on inaccurate shorter-range values, while max-based backups can amplify overestimation.
Idea. $\texttt{DCRL}$ recursively learns short-segment values before long-segment values using exact route factorization, then propagates across trajectories to recover optimal values.
Method
$\texttt{DCRL}$ makes the dependency between short- and long-segment values explicit through recursive divide-and-conquer learning.
Divide:Recursively split each segment at its midpoint until reaching single-step leaves.
Conquer:Learn values from leaves to root, so each parent is updated only after its children.
For a demonstrated route from state \(s\) to goal \(g\), $\texttt{DCRL}$ chooses its midpoint \(w\) and exactly factorizes its behavior value without selecting among noisy alternatives:
\[
V_\tau(s,g) = V_\tau(s,w) \cdot V_\tau(w,g).
\]
The balanced tree gives $\texttt{DCRL}$ logarithmic bootstrap depth in the horizon, compared with linear worst-case depth for standard backups.
Parallelize recursive value learning. Slot scheduling interleaves multiple trees, preserving child-to-parent ordering within each slot while maintaining diverse minibatches.
Recover optimal values. Exact factorization learns behavior values ($V_\tau$) along demonstrated routes, but optimal routes may require composing segments across trajectories. A separate $n$-step propagation objective performs this composition, refining behavior values toward optimality ($V^\ast$).
Experiments
1. How well does $\texttt{DCRL}$ compare with prior offline GCRL methods?
Long-Horizon Results
humanoidmaze-giant
puzzle-4x5
puzzle-4x6
cube-quadruple
cube-octuple
On the five hardest long-horizon OGBench tasks, $\texttt{DCRL}$ achieves the best average score, leads on $\texttt{humanoidmaze-giant}$ and both puzzle tasks, and is the only flat method with nonzero success on $\texttt{cube-octuple}$.
Standard-Horizon Results
Across 10 state-based and 10 pixel-based OGBench tasks, $\texttt{DCRL}$ achieves the best aggregate performance in both modalities.
CALVIN Results
Initial State
1. Open drawer
2. Turn on lightbulb
3. Move slider left
4. Turn on LED
On state-based CALVIN, $\texttt{DCRL}$ achieves the highest four-subtask completion rate (8%). On pixel-based CALVIN, $\texttt{DCRL}$ completes at least three subtasks in 8% of rollouts and all four in 1%, whereas $\texttt{HIQL}$, $\texttt{TRL}$, and $\texttt{TD-n}$ never reach the third subtask.
2. Is $\texttt{DCRL-n}$ better than $\texttt{TD-n}$ and $\texttt{TRL-n}$ across propagation and task horizons?
At $n\in\{1,5\}$, $\texttt{DCRL-n}$ performs similarly to $\texttt{TD-n}$ and below $\texttt{TRL-n}$; for $n\ge25$, it maintains a clear margin over both. $\texttt{DCRL-n}$ has logarithmic bootstrap depth in the task horizon $H$, versus linear worst-case depth for $\texttt{TD-n}$ and $\texttt{TRL-n}$, and exhibits much slower empirical Q-error growth.
3. Comparison with alternative divide-and-conquer strategies
We compare $\texttt{DCRL}$ with alternative strategies for organizing divide-and-conquer value learning:
- $\texttt{Curriculum}$ ($\Delta$): gradually increases the maximum segment length over $H/\Delta$ stages, where $\Delta$ is the segment-length increment per stage.
- $\texttt{TRL}$ ($\lambda$): up-weights short state-goal pairs and down-weights long ones using $w(s_i,s_j):=\bigl(1+\log_\gamma Q(s_i,a_i,s_j)\bigr)^{-\lambda}$.
- $\texttt{DCRL}$ (w/ random split): replaces the midpoint with a random in-trajectory split.
- $\texttt{DCRL}$ (w/ reverse order): processes each recursive tree from parents to children.
$\texttt{DCRL}$ reaches 93% success, over $1.7\times$ $\texttt{TRL}$'s success rate, at 24% lower throughput. Random splitting preserves performance but lowers throughput by creating deeper, less balanced trees. Reversing the learning order reduces success to 18%, consistent with parents bootstrapping from not-yet-updated children.
4. Ablation studies
We ablate the three core components of $\texttt{DCRL}$:
- $\texttt{DCRL}$ (w/o recursion): replaces recursive scheduling with i.i.d. sampling and adds one-step grounding.
- $\texttt{DCRL}$ (w/o propagation): uses recursive value learning alone to recover optimal values.
- $\texttt{DCRL}$ (w/ max-backup): replaces exact factorization with subgoal maximization.
Removing recursion causes the largest performance drop, highlighting the recursive tree's central role. Removing propagation also substantially hurts performance, supporting $\texttt{DCRL}$'s separation of recursive behavior-value learning and optimal-value propagation. Max-based backups reduce success and throughput, consistent with selection-induced overestimation and subgoal-search overhead.
Remarks
$\texttt{DCRL}$ demonstrates the potential of recursive value learning across diverse long-horizon goal-reaching tasks, while leaving two important directions for future work.
1. Extending to stochastic dynamics.
Like other triangle-inequality methods, $\texttt{DCRL}$ assumes deterministic dynamics. Under stochastic dynamics, exact factorization generally fails because the expectation of a product need not factorize (i.e., $\mathbb{E}[XY] \neq \mathbb{E}[X]\mathbb{E}[Y]$). Extending recursive sampling beyond this assumption remains an important direction.
2. Understanding when hierarchy helps.
Hierarchical methods outperform $\texttt{DCRL}$ on cube-manipulation tasks, although these tasks have shorter task horizons than $\texttt{humanoidmaze-giant}$, where $\texttt{DCRL}$ excels. Their difficulty therefore does not stem solely from horizon length. Understanding when hierarchy remains beneficial may clarify how the curse of horizon interacts with other sources of task difficulty in GCRL.
Citation
@article{jeon2026recursive,
title={Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL},
author={Jeon, Hyeonseong and Lee, Youngwoon},
journal={arXiv preprint arXiv:2609.02237},
year={2026}
}