Reasoning needs a way into action

A reasoning model can understand “put the yellow block on the orange one, then add the red block.” The robot policy still needs to know which objects those words refer to, where to move, and when to continue to the next step.

COMPASS is a SceneCode-based bridge system between System 2 and System 1. System 2 interprets the task and writes a structured program. SceneCode carries the objects, goals, steps, and expected outcomes. The bridge grounds these references in the scene and passes execution targets into System 1.

SYSTEM 2

Reason & plan

Understand the context.
Choose the next goal.

THE SHARED PROGRAM

SceneCode

objectsgoalsstepschecks
SYSTEM 1

Act in the world

Follow grounded targets
through learned embeddings.

COMPASS connects the reasoning model and action policy through SceneCode.

What is SceneCode?

SceneCode is a structured description of the task and the scene that both systems use. It names the objects, records their current states and relations, specifies the next action segment, and states what should be true afterward.

The task program carries the longer-term requirement: for example, the orange block must be below the yellow block, and the red block must finish on top. At each review, System 2 updates the current scene and the commands needed to make progress toward that requirement.

Objects & state

What is present, where it is, and what is currently held.

orange_block · anchor [325, 254] · held by left gripper
Relations

How objects and the robot relate to each other.

above(yellow_block, orange_block)
Commands

What should happen during the next action segment.

steer → orange_block · object = yellow_block · release = true
Expected outcomes

What the next observation should establish.

on_top(yellow_block, orange_block)

An object reference includes an image anchor, such as a pixel near the center of the orange block. SAM3 uses that reference to locate the object. Depth and camera geometry lift its location into 3D, giving the controller a target that refers to the current scene.

The following is the task-level plan from the recorded color-order stacking case. In the case studies below, the scene description, commands, and expected outcomes change as execution proceeds.

SceneCode · recorded execution
{
  "program": [
    "Use the orange block as the tower base.",
    "Place the yellow block centered on the orange block.",
    "Place the red block centered on the yellow block.",
    "Return the active robot arm to its reset pose."
  ],
  "goal_state": [
    "on_top(yellow_block, orange_block)",
    "on_top(red_block, yellow_block)",
    "upright_stack(orange_block, yellow_block, red_block)",
    "reset(active_robot_arm)"
  ]
}
A compact excerpt from the color-order stacking case.

The complete pipeline

COMPASS carries the task through program construction, scene grounding, target encoding, policy execution, and outcome checking. Each new observation can change the next command while the original task requirement stays explicit.

01Instruction or demonstration

Task context

Objects, rules, and the intended outcome.

02Program construction

System 2 → SceneCode

Write the scene, subgoals, commands, and checks.

03SAM3 + RGB-D

Ground the objects

Resolve image anchors into masks and metric targets.

04Learned bridge

Encode the target

Combine target geometry, robot state, and visual features.

05Frozen System 1

Execute with pi0.5

Inject the bridge vector and produce an action chunk.

06Execution feedback

Observe and revise

Compare the new scene with expect and update the program.

New observations → System 2 → updated SceneCode → next action
The program is carried through the full task loop, from context to checked execution.

There are two update rates. System 1 produces action chunks from the current observations and execution target. System 2 reviews the accumulated outcome at selected points and updates SceneCode. This lets the program retain a multi-step plan while responding to changes in the scene.

How the encoder connects the program to actions

The encoder is the learned connection from SceneCode’s grounded execution targets into pi0.5. It receives target geometry, the current robot state and execution phase, together with visual features from the policy. Its output changes the policy’s internal representation before actions are generated.

GROUNDED EXECUTION FEATURES

Target + robot state

Target point and hand-to-target offset
Hand / joint state · active arm · execution phase

CURRENT OBSERVATION

Pooled pi0.5 visual features → learned projection

→
TRAINABLE · ≈3.8M PARAMETERS
768tanh
→
768tanh
→
2048bridge vector

Visual and geometric features meet in the first hidden layer.

FROZEN PI0.5 · PALIGEMMA

Inject into the language-token residual

h10 ← h10 + α · eϕ(f, v)

The action expert then generates the robot’s action chunk.

Layer 10
Default MLP interface: two 768-unit hidden layers and a 2,048-dimensional output. The policy weights stay frozen.

The default network is an MLP with two hidden layers of width 768, a visual-feature projection, and a 2,048-dimensional output. The resulting vector is added to the language-token residual at PaliGemma layer 10. The target features include the hand-to-target displacement; RoboDojo also supplies the arm state and active-arm information needed for bimanual execution.

Learning the bridge

Training runs the encoder and frozen policy together. The policy predicts an action chunk, and the training objective measures how closely its resulting hand or object position follows the grounded target. Reach and carry terms encourage progress toward the goal; gripper and orientation terms retain the associated manipulation behavior. Release targets add the opening condition.

Gradients pass through pi0.5 to train the encoder. The reasoning model and pi0.5 weights stay frozen. Teacher corrections collected on states encountered during execution can be added as DAgger training data for this bridge.

Planning, correction, and embedding work together

01 / PLAN

Decide what comes next.

Keep the selected objects, subgoals, order, and completion conditions in the same program.

02 / CORRECTION

Redirect execution.

Check what happened, revise the target or plan, and continue. Teacher corrections also supply DAgger training data for the bridge encoder.

03 / EMBEDDING

Give intent a location.

Ground objects with SAM3 and depth. Encode the target, robot state, and policy features into a signal that guides System 1.

The default system uses Sol for reasoning and pi0.5 for robot actions. SceneCode records the plan and checks; perception resolves its object references; the encoder supplies target guidance; and runtime feedback determines which part of the program to execute next.

Case studies: SceneCode through execution

The following case studies show successive recorded reviews. Each keyframe is paired with the current SceneCode command and expected outcomes, so the progression can be read from top to bottom. Full synchronized recordings are linked below each case.

CASE STUDY / Planning

Keep the intended order while the scene changes.

The task fixes the final color order. The recorded reviews change the immediate subgoal: first approach the yellow block, then place the grasped orange block as the base, and then guide the yellow block onto it.

Recorded execution at 0.00 seconds
0.00s · recorded review

The orange block is available as the tower base, but no blocks are stacked yet.

SCENECODE UPDATE

{
  "next_command": {
    "op": "steer",
    "arm": "right",
    "target": "yellow_block",
    "release": false,
    "segments": 2,
    "instruction": "Pick up the yellow block."
  },
  "expect": [
    "right gripper holds yellow_block",
    "orange_block remains on table",
    "red_block remains on table"
  ]
}
Recorded execution at 6.00 seconds
6.00s · recorded review

The orange block has been grasped and moved to the intended tower area, but it still must be placed as the base.

SCENECODE UPDATE

{
  "next_command": {
    "op": "native",
    "instruction": "Put the orange block down as the tower base.",
    "segments": 4
  },
  "expect": [
    "orange_block is released upright on the table",
    "orange_block is the base location for the tower",
    "left gripper is empty"
  ]
}
Recorded execution at 9.60 seconds
9.60s · recorded review

The orange block is positioned as the base, and the right arm is holding the yellow block above it.

SCENECODE UPDATE

{
  "next_command": {
    "op": "steer",
    "arm": "right",
    "target": "orange_block",
    "object": "yellow_block",
    "release": true,
    "segments": 3,
    "instruction": "Place the yellow block on the orange block."
  },
  "expect": [
    "on_top(yellow_block, orange_block)",
    "held_by(red_block, right_gripper)"
  ]
}

Watch the full execution with SceneCode alongside ↗

In the stacking case, the first commands request the yellow block. At 6.00 s, the review describes the orange block as held and changes the immediate instruction to placing that block as the base. At 9.60 s, the program describes the yellow block above the orange base and issues a steer command targeting the orange block, with the yellow block as the held object and release: true.

CASE STUDY / Memory

Remember a color order after the objects disappear.

Three cups cover the colored blocks. SceneCode retains the required red → green → blue uncovering order while updating which cups and blocks are currently visible.

Recorded execution at 0.00 seconds
0.00s · recorded review

No task actions are yet achieved; the three blocks and three matching yellow cups are visible in their left-to-right positions.

SCENECODE UPDATE

{
  "next_command": {
    "op": "native",
    "segments": 6
  },
  "expect": [
    "covered_by(green_block,left_yellow_cup) may be initiated or achieved",
    "covered_by(blue_block,middle_yellow_cup) may be initiated or achieved",
    "covered_by(red_block,right_yellow_cup) may be initiated or achieved"
  ]
}
Recorded execution at 10.80 seconds
10.80s · recorded review

All three cups appear placed over their matching blocks from left to right, so the covering phase is complete and the red cup should now be removed.

SCENECODE UPDATE

{
  "next_command": {
    "op": "native",
    "segments": 6
  },
  "expect": [
    "covered_by(green_block,left_yellow_cup)",
    "covered_by(blue_block,middle_yellow_cup)",
    "covered_by(red_block,right_yellow_cup)",
    "uncovered(red_block)"
  ]
}
Recorded execution at 18.00 seconds
18.00s · recorded review

The middle blue block is covered, but the left-to-right covering phase is not yet correct because the left cup is on the red block and the green block remains uncovered.

SCENECODE UPDATE

{
  "next_command": {
    "op": "native",
    "segments": 6
  },
  "expect": [
    "covered_by(green_block,left_yellow_cup)",
    "covered_by(blue_block,middle_yellow_cup)",
    "covered_by(red_block,right_yellow_cup)"
  ]
}

Watch the full execution with SceneCode alongside ↗

The cover-and-uncover task separates current visibility from the remembered requirement. The longer-term plan retains the color order; scene updates record the covering phase, the cups already placed, and the next object to uncover. The live checks show what the controller expects to establish before continuing.

CASE STUDY / Coordination

Carry one plan across several objects and destinations.

The mouse, keyboard, figurine, clock, and leftover objects have different destinations. The program tracks completed placements and selects what remains to be done.

Recorded execution at 0.00 seconds
0.00s · recorded review

No task-program step is yet complete; the policy should begin by placing the red mouse on the black mouse pad.

SCENECODE UPDATE

{
  "next_command": {
    "op": "native",
    "segments": 6
  },
  "expect": [
    "on(red_mouse, black_mouse_pad) or active progress toward placing the mouse on the pad)"
  ]
}
Recorded execution at 7.20 seconds
7.20s · recorded review

The alarm clock has been placed on top of the drawer unit; the mouse, keyboard, figurine, drawer opening, and miscellaneous-item storage remain to be completed.

SCENECODE UPDATE

{
  "next_command": {
    "op": "native",
    "segments": 6
  },
  "expect": [
    "The policy continues the trained organization sequence and begins or advances placement of the mouse, keyboard, or figurine without disturbing the alarm clock."
  ]
}
Recorded execution at 14.40 seconds
14.40s · recorded review

The mouse is on the mouse pad, and the policy is actively pushing the keyboard toward the outlined frame; later ordered steps remain.

SCENECODE UPDATE

{
  "next_command": {
    "op": "native",
    "segments": 6
  },
  "expect": [
    "on(red_mouse, black_mouse_pad) remains true",
    "white_keyboard moves farther into keyboard_frame",
    "policy continues the keyboard-placement step"
  ]
}

Watch the full execution with SceneCode alongside ↗

Workspace organization needs several object–destination pairs. SceneCode preserves those assignments while the scene changes after each placement. The recorded updates make it possible to inspect which subgoals the controller considers complete and which command it chooses next.

The linked recordings show the execution and changing SceneCode side by side.

Cases: different tasks, the same bridge

Ordering, memory, object selection, precision, and coordination put different demands on the robot. These recorded COMPASS executions show how the task requirements become concrete actions. Each example includes a link to its execution video.

A brief look at the results

Public benchmarks test whether the system follows changes in goals, instructions, and positions. COMPASS reaches 83.5% on LIBERO-OOD and 50.1% on LIBERO-CF.

Benchmarkpi0.5COMPASSEarlier method · paper result
LIBERO-OOD38.8%83.5%TLI · 83.0% ↗
LIBERO-CF15.7%50.1%CAG-VA · 21.7% ↗
LangGap25.7%60.7%—
PRO-position30.0%42.5%—
PRO-task25.0%41.5%—

Task success (%). Paper-method names link to the original result tables.

On RoboDojo ICL, the current run records a task-mean progress score of 17.69 for COMPASS and 15.90 for pi0.5, with 79 versus 65 successful cases.

Switching only the encoder off gives the most direct comparison. On 250 paired LIBERO-CF cases that reuse the same saved Sol plans, COMPASS succeeds in 126; without encoder injection it succeeds in 36. Without the encoder, the robot finished a different, familiar task in 143 episodes, against 20 with it.

LIBERO-CF suiteCasesCOMPASSWithout encoder injection
Spatial7538 (50.7%)17 (22.7%)
Object5018 (36.0%)2 (4.0%)
OOD7542 (56.0%)5 (6.7%)
Long5028 (56.0%)12 (24.0%)
All250126 (50.4%)36 (14.4%)

Successes on the same cases, plans, seeds and controller; only the encoder injection differs.

What SceneCode makes possible

The core of COMPASS is a shared representation that reasoning can write, perception can ground, and the action policy can use. Planning, corrections, and embeddings work through that bridge as the task unfolds.

Explore the code ↗