Skip to main content
run-policy --policy_type custom and collect-dagger --policy_type custom use this interface to talk to your policy endpoint. The almond_axol.policy SDK’s serve() implements the endpoint side for you. Read on to implement it yourself, for example in another language. The handshake carries version: 2. The robot rejects any other version and never downgrades. Each side owns different work:
  • Your endpoint owns the model, the instruction text, normalization, conditioning and its cache of predictions.
  • The robot owns camera capture, the dispatch clock, and replacing future targets when a new plan arrives.
Every reply contains absolute targets in the agreed action layout, units and robot frames. The robot checks names and shapes. Matching units and frame conventions is up to your endpoint. The observation and action layouts can differ, for example joint observations with Cartesian actions.

Setup

PlanPolicyClient.connect(spec) sends hello. The endpoint replies with ready, which echoes the whole validated spec. The client refuses any mismatch.
hello and ready both contain:
  • protocol: "axol-policy" and version: 2,
  • the ordered state and action names,
  • the cameras, each with name, shape: [height, width, 3] after preparation, and codec: "png",
  • the four scheduling fields above.
Your endpoint must agree with the robot’s camera preparation and action names, and reject an incompatible spec before it sends ready. The robot client sends dispatch_feedback: true in hello, and ready must echo it. This turns on the last_dispatched reference described below. PlanSpec.dispatch_feedback can be false for low-level clients that don’t support it. Those clients leave the capability and the last_dispatched field off the wire. An explicit dispatch_feedback: false is invalid, so leave the field out instead. The robot client and the endpoint must agree on this capability, and the handshake never downgrades it. max_adoption_offset_steps is the oldest a reply may be, in control steps from its request, when the robot starts using it. It never delays an early reply. The wire format allows None to remove this deadline, and rows still expire at their scheduled times. The robot runtime requires an integer within the horizon. Every image must be PNG, so RGB pixels arrive unchanged after preparation.

Recurring request and reply

Each WebSocket message is binary: a big-endian uint32 byte count, that many bytes of UTF-8 JSON header, then the binary payload. WebSocket compression is off, because the images are already PNG. An infer header is:
The payload is the complete PNG files, one after another, in the agreed camera order. Each decodes to an RGB uint8 array. Timestamps are integer nanoseconds on the robot’s monotonic clock, so don’t compare them with the desktop’s clock. The robot checks camera count, order and shape, PNG format, byte lengths, finite state values and every message field.

continuation

continuation names the prediction the robot is executing and the first row it hasn’t run yet. The queued targets are always an unchanged suffix of that prediction. It is null at the start and after a reset. If your cache doesn’t have that prediction, or the suffix is past its end, reject the request rather than conditioning on a different plan. The robot never blends or edits queued targets, which is what keeps this reference exact. Filtering and low-level control happen after the targets.

last_dispatched

When dispatch_feedback is on, every infer header contains last_dispatched. Its value is null or an object with prediction_id and a zero-based row. It names the last published row whose send on the robot completed in the current reset generation. When dispatch_feedback is off, leave the field out. last_dispatched is not:
  • a hardware acknowledgement,
  • a measured pose or proof that the arm tracked the target,
  • the setpoint after trajectory filtering or IK.
It names the exact row your endpoint sent, before the robot’s filtering. The client records both references when it makes the request, using the latest completed send. A send still in progress or a failed send is never reported. A send from before a reset never fills in the reference afterwards. null means there is no completed send in the current reset generation, including during recovery. It doesn’t mean no command was sent, or that the robot has stopped. The two references can name different plans. For example, after it sends X[12], the client may adopt a late plan Y whose next row is Y[3]. Before it sends that row, an observation reports continuation={prediction_id: "Y", from_row: 3} and last_dispatched={prediction_id: "X", row: 12}. Y[2] was skipped, so continuation.from_row - 1 doesn’t name the previous command. If your endpoint needs this history, keep both plans.

delay_steps

By default the robot sends no latency estimate. An adaptive endpoint can opt into the optional top-level integer delay_steps. It is input for conditioning, not a request to delay adoption. The robot caps it at actions_per_chunk, which means a full horizon or more. Instruction text stays on the desktop.

The actions reply

The actions reply header contains:
The payload is row-major, little-endian float32 values. An optional max_adoption_offset_steps can tighten the session deadline for this reply. The wire carries no robot tick numbers. The robot ties row zero to the request’s start time, which it saved locally. It drops rows whose time has passed or that are already committed, and starts the rest right away.

Lifecycle and failure

After connecting, call client.reset(episode) before the first inference. reset carries an integer episode, and reset_ok echoes it. Before it replies, the endpoint must clear every cached prediction and all recurrent state. Request IDs are non-empty strings, unique within an episode. A client has at most one request outstanding. On a hold, a task change or a new episode, the robot drops pending predictions right away. It then sends reset after any outstanding request, so a reply from before the reset can never refill the queue. Change the desktop’s instruction text at this boundary too. When a reply is late or the plan runs out, the robot follows its late_policy:
  • "blocking_refresh" (the default) drops the current plan and dispatch reference. Once pending inference finishes, it resets the endpoint, holds, and requests a fresh plan with no continuation. It runs up to request_interval rows of that plan, then refreshes again the same way. It stays in this mode until the scheduler itself is reset, and doesn’t go back to overlapping requests on its own.
  • "abort" ends the session.
An error reply has type: "error" and a message. The endpoint must reject malformed requests, unknown continuation or last_dispatched references, and invalid model output. The client closes the connection on an error reply, a malformed response or a timeout. After a transport failure, open a new connection and reset it, so a late reply can’t be taken for a newer request. The client doesn’t reconnect or resume motion on its own.

Robot control loop

Select the interface with axol run-policy --policy_type custom --server_host HOST --server_port 8765 --task TASK. The normal robot and camera settings still apply. task labels the recording. The robot doesn’t send it to the endpoint, which chooses its own instruction. The robot captures observations and talks to the endpoint outside the control loop, so encoding and network waits never delay a motor command. It keeps one request outstanding and one adopted plan, with no queue of future plans. A row that is being sent can’t be replaced. A request saves the next dispatch tick as its start. If that start is o and the next free tick when the reply arrives is k, the robot adopts the reply from row max(0, k - o). Row zero is never shifted to the reply’s arrival time. At the start there is no plan, so the robot holds and the first reply starts at row zero. Image preparation, encoding and transport all count against the request’s adoption deadline. A reused camera frame keeps its original capture time, and the robot checks freshness before inference. If the control loop misses a deadline, the robot drops the plan instead of sending a burst of missed targets. For example, at 30 Hz with a 30-row horizon and a request every 10 rows: Each plan’s identifier is the request_id of the request it answered. The example assumes every send has completed before the next request. A send still in progress is not yet reported in last_dispatched. Custom policies skip LeRobot’s chunk alignment and temporal ensembling. Their targets still go through joint limits, Cartesian IK when configured, the configured trajectory filter, the realtime controller and the configured contact safeguards. robot_config.action_space selects joint or cartesian actions without changing the observation layout. If you leave it out, the action layout follows observe_cartesian.

Supporting different policy families

Because the robot executes one unchanged suffix, (prediction_id, from_row) fully describes the queued targets. An endpoint that keeps the plans it published can rebuild that suffix without the robot sending the action matrix back each time. Keep both plans when continuation and last_dispatched name different IDs. Convert relative actions to absolute targets in the agreed layout before you publish them. Preprocessing, normalization and recurrent state stay on the endpoint.
  • Real-time chunking. Condition the next prediction on the remaining suffix, with a fixed latency budget or the optional delay_steps. The robot keeps running the old plan during inference, then skips the rows of the new plan whose time has passed. It doesn’t need to know whether you used hard prefix conditioning, inpainting or another method.
  • Temporal ensembling. Keep overlapping raw predictions on the endpoint, compute the weighted targets, and publish the result. References name the published targets, not a raw model chunk. The robot never stores weights or re-ensembles a reply. request_interval=1 allows a request on every control tick, but a per-tick ensemble also needs inference fast enough to keep up.
The interface defines how plans are executed and reported. It doesn’t make arbitrary replies smooth. Your endpoint must produce targets that join up at plan boundaries and follow the agreed layout and timing. The robot’s filters and limits act on the motor commands after the targets, and last_dispatched doesn’t report their output. Timestamped state is the physical feedback. Smooth motion across plans and plan history after a recovery are not guaranteed. In "blocking_refresh" mode the reset clears that history.

DAgger ownership and recording

collect-dagger --policy_type custom --hold_to_intervene true uses the same messages and scheduler. One control loop sends both the policy’s and the operator’s commands.
  • Holding either grip cancels pending policy work and gives the operator that arm. The other arm holds.
  • If the operator’s link drops while grips are held, the operator keeps control and the arm holds its last command.
  • With shadow_inference=true, observations and inference continue during operator control, but the robot discards the replies.
  • Releasing both grips starts a new generation. The robot resets the endpoint after any outstanding reply, and holds the operator’s last command until a fresh plan arrives. A joint-space handover after IK starts from that held command, with limited velocity and acceleration.
  • Grabbing a grip again cancels that handover too. A late prediction from before an intervention can never take control back.
Recording continues in the same dataset episode, with a per-row intervention label and capture fences where control changes hands.
  • record_joint_actions=true records the resolved joint commands, whatever the policy’s action layout.
  • home_on_start=true homes the arms before the first episode.
  • start_from_current_pose=true keeps the pose set during scene reset for later episodes.
  • Pressing q in the terminal, idle or mid-episode, parks the arms through rest and zero before it turns off the motors.
Contact stops, faults and interrupted cleanup keep the arms supported.