RoboCasa LoadDishwasher
Only method
to converge in the OGPO comparison
VaDar: Implicit Goal Conditioning
via Value Disaggregation
1 Carnegie Mellon University · * Equal contribution
Work supported by a TRI University 3.0 grant.
VaDar converts a sparse task reward into per-subgoal rewards, each of which provides signal as soon as its subgoal is completed. So each transition is used to update a shared critic across multiple temporal resolutions. We hypothesize that this enables an implicit Bellman backup in the critic network activations (more below). Through careful experiments, we show that VaDar is always competitive with goal-conditioned, dense-reward and sparse-reward baselines, and often much better. As an added benefit, its policy needs no subgoal labels at test time.
Introduction
Learning long-horizon tasks with RL
Learning to solve long-horizon tasks via RL is a fundamental challenge in many domains, especially robotics. Many long-horizon robot manipulation tasks can be decomposed into subtasks.
Robomimic ToolHang · four subtasks
Conventional wisdom suggests leveraging this natural subgoal decomposition to solve these tasks efficiently. The prevalent methods that exploit this subgoal structure (which we term hierarchical goal conditioning, HGC) learn a subgoal-conditioned policy and rely on a dedicated planner, often a VLM, to determine which subgoal is active at inference time. The policy is then conditioned on the active subgoal to solve the task.
For current robotic manipulation tasks, VLMs are increasingly good at detecting the active subtask and at supplying completion signals that tell whether a subtask has been accomplished.
We first look at the pitfalls of conventional HGC methods, then modify them to arrive at our proposed method, Value Disaggregation (VaDar).
The first drawback is determining the current active subgoal while the policy is running. This requires a dedicated planner or querying the VLM during inference, which is both expensive and slow. An ideal method would use these expensive VLM calls only at training time, when we can spend the resources, and keep inference cheap and reliable.
The more pressing drawback is that simply conditioning the policy on the current active subgoal leads to myopic downstream behavior. This is severe in many robotic manipulation tasks, especially when future subgoals depend on the current one. How a tool will be used dictates how it should be grasped. A policy conditioned only on “grasp the tool”, with no awareness of what comes next, can complete the grasp in a way that makes the next step impossible. Hence we need a mechanism to overcome the myopia that comes from conditioning the policy on the current subgoal alone. Note that this challenge would persist even with VLAs capable of acting and reasoning jointly.
Summarising this discussion, the two desiderata we need in our algorithm are:
- Desideratum 1: no subgoal dependence in the policy, to avoid costly VLM calls during inference.
- Desideratum 2: no myopic behavior; the policy should receive information about future subgoals.
Value-based RL
Learning critics
In value-based RL we learn a critic : how much return we expect if we take action in state . The policy then picks actions the critic likes. We train the critic with temporal-difference (TD) learning, regressing it toward the reward plus its own guess at the next state:
With only a sparse reward at the end of the task, this signal has to travel backward one state at a time, so the critic needs good values near the end before it can learn anything near the start. With subgoals, the critic can also take a subgoal as input, , and learn the return for reaching that subgoal.
Implicit hierarchical goal conditioning (iHGC)
We can take the subgoal out of the policy and keep it only in the critic. We refer to this method as iHGC (implicit hierarchical goal conditioning). This satisfies desideratum 1, and in our experiments iHGC already beats HGC in some cases. But the policy is still not aware of future subgoals, and iHGC falls prey to the same myopia we see in HGC. This brings us to desideratum 2.
PreSoakPan
Future awareness: VaDar
Our method needs to be aware of future subgoals to curb the inherent myopia of naive goal conditioning, while keeping the subgoal dependence only on the critic side.
VaDar keeps the subgoals in the critic, like iHGC, but the critic now predicts a value for every subgoal at the same time, not just the active one. Each subgoal gets its own reward and its own TD target, and we sum the losses to update one shared critic.
One sampled transition Feed the same state and action to every prediction
↓
Shared critic representation
Four predicted subgoal returns from the same transition
Grasp frame
TD target
Insert frame
TD target
Grasp wrench
TD target
Hang wrench
TD target
↑Add the four lossesUpdate the shared critic
The four subgoals are from ToolHang. Each target uses that subgoal's reward and next-state value.
To train the policy, we combine these predictions into one objective:
The weights (the gating) decide which subgoal values the policy is trained on. We try three gatings. Current-myopic uses only the subgoal the robot is working on, which is what HGC and iHGC do. Current-future adds every later subgoal. Next-future, our default, drops the current subgoal and uses only the later ones; at the last subgoal it uses the final value.
ToolHang
Which values train the policy?
Trains the critic (all subgoals)
Trains the policy (selected subgoals)
Each stage is a different moment in the task. The dots are training signals, not robot motion.
So how does VaDar meet both desiderata? Every transition trains the critic on every subgoal, so the critic gets far more signal than from the final reward alone (desideratum 1). And the policy is trained on the values of future subgoals, so grasping the tool in a way that ruins the next step gets a low score (desideratum 2). As an added benefit, the policy only takes the state, so it needs no subgoal labels at test time.
What happens to finished subgoals?
Once a subgoal is done, its value should stop growing, so we clip its TD target. The clipping rule and the extension to subgoals that form a directed acyclic graph are in the paper.
Results
Does our method help us reliably learn long-horizon robot manipulation?
VaDar improves learning on long-horizon manipulation tasks
RoboCasa PreSoakPan
~2×
the success of the strongest baseline (iHGC)
Watch VaDar rollouts
PreSoakPan
ArrangeTea
ToolHang
We start every method from the same behavior-cloned policy with low success and fine-tune online, using the same base algorithm within each comparison. With OGPO on RoboCasa, VaDar is the only method that converges on LoadDishwasher, and on PreSoakPan it reaches nearly twice the success of the strongest baseline, iHGC. With FQL it learns faster on Transport and ToolHang. On OGBench with SAC it matches the goal-conditioned baselines. Across all of these, VaDar is always competitive with goal conditioning and often better.
Understanding VaDar in depth
Multi-resolution temporal supervision gives an implicit Bellman backup
To test whether the intermediate targets improve the critic by themselves, we train Last-Q: its critic is supervised on every subgoal, but its policy is trained only on the final-subgoal value, the same policy objective as the sparse-reward baseline. Last-Q substantially outperforms the sparse baseline on RoboCasa and approaches VaDar on several tasks, so the intermediate targets improve the final-subgoal critic even though the policy never uses them.
Isn't this just reward shaping? Theoretically, yes, but only theoretically. With fixed weights and exact Q-functions under the same policy, a weighted sum of subgoal values is the value of the weighted sum of subgoal rewards. VaDar breaks both assumptions: its weights change with the state, and its critics are clipped and learned. So we test it. VaDar-Shaped adds up the subgoal rewards before the Bellman backup and trains one scalar critic on them. It fails to learn on the harder RoboCasa tasks and is much less sample efficient elsewhere. Together with Last-Q, this says VaDar's gains come from supervising the critic at multiple temporal resolutions, not from reward shaping alone.
So VaDar lets us learn a good critic?
We freeze a behavior-cloned policy and train only the critic. Then we ask: does the critic pick the actions that lead to success? At test time the frozen policy proposes action chunks and the critic picks the one it values most (best-of- steering, as in Q-chunking). We compare a VaDar critic with a Sparse critic trained on the same data.
ArrangeTea
ToolHang
ArrangeTea
23
%
→
55
%
No steering → VaDar critic
Sparse critic: 23%
ToolHang
32
%
→
77
%
No steering → VaDar critic
Sparse critic: 44%
Mean success after 280K critic-training steps with N = 4. Three seeds, 96 episodes each.
Steering uses subgoal-completion labels at test time to compute the next-future weights.
Beyond long horizon
The subgoals for value disaggregation need not be subtasks of a long-horizon task.
Value disaggregation also supports multi-objective learning
Here the subgoals are two objectives: push the T onto the target, and stay away from the obstacle. VaDar learns one value for each and adds them with a safety weight . Agg-SAC adds the rewards instead and learns a single critic. At small the two do the same. At and , Agg-SAC's success drops to nearly zero, while VaDar keeps learning.
Extensions beyond sequential subgoals
The subgoals don't have to be a chain; VaDar also works when they form a directed acyclic graph. The labels don't have to come from the simulator either: on PreSoakPan, VLM labels reach similar final success to oracle labels, with a modest change in sample efficiency.
VaDar does assume a fixed set of subgoals. Handling continuous or very large goal spaces, like goals given as images or text, is still open.
This page covers the main story. For the full method, the theory and the rest of the experiments, read the full paper on arXiv ↗.
Citation
@article{saxena2026vadar,
title = {VaDar: Implicit Goal Conditioning via Value Disaggregation},
author = {Saxena, Shashwat and Goel, Mehul and Venkataraman, Sreyas and Patil, Sarvesh and Simchowitz, Max},
journal = {arXiv preprint arXiv:2610.26629},
year = {2026}
}