run-policy --policy_type custom and collect-dagger --policy_type custom use
this interface to talk to your policy endpoint. The
almond_axol.policy SDK’s serve() implements the endpoint
side for you. Read on to implement it yourself, for example in another
language.
The handshake carries version: 2. The robot rejects any other version and
never downgrades.
Each side owns different work:
- Your endpoint owns the model, the instruction text, normalization, conditioning and its cache of predictions.
- The robot owns camera capture, the dispatch clock, and replacing future targets when a new plan arrives.
Setup
PlanPolicyClient.connect(spec) sends hello. The endpoint replies with
ready, which echoes the whole validated spec. The client refuses any
mismatch.
hello and ready both contain:
protocol: "axol-policy"andversion: 2,- the ordered state and action names,
- the cameras, each with
name,shape: [height, width, 3]after preparation, andcodec: "png", - the four scheduling fields above.
ready.
The robot client sends dispatch_feedback: true in hello, and ready must
echo it. This turns on the last_dispatched reference described below.
PlanSpec.dispatch_feedback can be false for low-level clients that don’t
support it. Those clients leave the capability and the last_dispatched field
off the wire. An explicit dispatch_feedback: false is invalid, so leave the
field out instead. The robot client and the endpoint must agree on this
capability, and the handshake never downgrades it.
max_adoption_offset_steps is the oldest a reply may be, in control steps
from its request, when the robot starts using it. It never delays an early
reply. The wire format allows None to remove this deadline, and rows still
expire at their scheduled times. The robot runtime requires an integer within
the horizon. Every image must be PNG, so RGB pixels arrive unchanged
after preparation.
Recurring request and reply
Each WebSocket message is binary: a big-endian uint32 byte count, that many bytes of UTF-8 JSON header, then the binary payload. WebSocket compression is off, because the images are already PNG. Aninfer header is:
continuation
continuation names the prediction the robot is executing and the first row
it hasn’t run yet. The queued targets are always an unchanged suffix of that
prediction. It is null at the start and after a reset. If your cache doesn’t
have that prediction, or the suffix is past its end, reject the request rather
than conditioning on a different plan. The robot never blends or edits queued
targets, which is what keeps this reference exact. Filtering and low-level
control happen after the targets.
last_dispatched
When dispatch_feedback is on, every infer header contains
last_dispatched. Its value is null or an object with prediction_id and a
zero-based row. It names the last published row whose send on the robot
completed in the current reset generation. When dispatch_feedback is off,
leave the field out.
last_dispatched is not:
- a hardware acknowledgement,
- a measured pose or proof that the arm tracked the target,
- the setpoint after trajectory filtering or IK.
null means there is no completed send in the current reset generation,
including during recovery. It doesn’t mean no command was sent, or that the
robot has stopped.
The two references can name different plans. For example, after it sends
X[12], the client may adopt a late plan Y whose next row is Y[3]. Before
it sends that row, an observation reports
continuation={prediction_id: "Y", from_row: 3} and
last_dispatched={prediction_id: "X", row: 12}. Y[2] was skipped, so
continuation.from_row - 1 doesn’t name the previous command. If your
endpoint needs this history, keep both plans.
delay_steps
By default the robot sends no latency estimate. An adaptive endpoint can opt
into the optional top-level integer delay_steps. It is input for
conditioning, not a request to delay adoption. The robot caps it at
actions_per_chunk, which means a full horizon or more. Instruction text
stays on the desktop.
The actions reply
The actions reply header contains:
max_adoption_offset_steps can tighten the session deadline for this reply.
The wire carries no robot tick numbers. The robot ties row zero to the
request’s start time, which it saved locally. It drops rows whose time has
passed or that are already committed, and starts the rest right away.
Lifecycle and failure
After connecting, callclient.reset(episode) before the first inference.
reset carries an integer episode, and reset_ok echoes it. Before it
replies, the endpoint must clear every cached prediction and all recurrent
state. Request IDs are non-empty strings, unique within an episode. A client
has at most one request outstanding.
On a hold, a task change or a new episode, the robot drops pending
predictions right away. It then sends reset after any outstanding request,
so a reply from before the reset can never refill the queue. Change the
desktop’s instruction text at this boundary too.
When a reply is late or the plan runs out, the robot follows its
late_policy:
"blocking_refresh"(the default) drops the current plan and dispatch reference. Once pending inference finishes, it resets the endpoint, holds, and requests a fresh plan with no continuation. It runs up torequest_intervalrows of that plan, then refreshes again the same way. It stays in this mode until the scheduler itself is reset, and doesn’t go back to overlapping requests on its own."abort"ends the session.
type: "error" and a message. The endpoint must reject
malformed requests, unknown continuation or last_dispatched references, and
invalid model output. The client closes the connection on an error reply, a
malformed response or a timeout. After a transport failure, open a new
connection and reset it, so a late reply can’t be taken for a newer request.
The client doesn’t reconnect or resume motion on its own.
Robot control loop
Select the interface withaxol run-policy --policy_type custom --server_host HOST --server_port 8765 --task TASK. The normal robot and
camera settings still apply. task labels the recording. The robot doesn’t
send it to the endpoint, which chooses its own instruction.
The robot captures observations and talks to the endpoint outside the control
loop, so encoding and network waits never delay a motor command. It keeps one
request outstanding and one adopted plan, with no queue of future plans. A row
that is being sent can’t be replaced.
A request saves the next dispatch tick as its start. If that start is o and
the next free tick when the reply arrives is k, the robot adopts the reply
from row max(0, k - o). Row zero is never shifted to the reply’s arrival
time. At the start there is no plan, so the robot holds and the first reply
starts at row zero.
Image preparation, encoding and transport all count against the request’s
adoption deadline. A reused camera frame keeps its original capture time, and
the robot checks freshness before inference. If the control loop misses a
deadline, the robot drops the plan instead of sending a burst of missed
targets.
For example, at 30 Hz with a 30-row horizon and a request every 10 rows:
Each plan’s identifier is the
request_id of the request it answered. The
example assumes every send has completed before the next request. A send
still in progress is not yet reported in last_dispatched.
Custom policies skip LeRobot’s chunk alignment and temporal ensembling. Their
targets still go through joint limits, Cartesian IK when configured, the
configured trajectory filter, the realtime controller and the configured
contact safeguards. robot_config.action_space selects joint or cartesian
actions without changing the observation layout. If you leave it out, the
action layout follows observe_cartesian.
Supporting different policy families
Because the robot executes one unchanged suffix,(prediction_id, from_row)
fully describes the queued targets. An endpoint that keeps the plans it
published can rebuild that suffix without the robot sending the action matrix
back each time. Keep both plans when continuation and last_dispatched name
different IDs. Convert relative actions to absolute targets in the agreed
layout before you publish them. Preprocessing, normalization and recurrent
state stay on the endpoint.
- Real-time chunking. Condition the next prediction on the remaining
suffix, with a fixed latency budget or the optional
delay_steps. The robot keeps running the old plan during inference, then skips the rows of the new plan whose time has passed. It doesn’t need to know whether you used hard prefix conditioning, inpainting or another method. - Temporal ensembling. Keep overlapping raw predictions on the endpoint,
compute the weighted targets, and publish the result. References name the
published targets, not a raw model chunk. The robot never stores weights or
re-ensembles a reply.
request_interval=1allows a request on every control tick, but a per-tick ensemble also needs inference fast enough to keep up.
last_dispatched
doesn’t report their output. Timestamped state is the physical feedback.
Smooth motion across plans and plan history after a recovery are not
guaranteed. In "blocking_refresh" mode the reset clears that history.
DAgger ownership and recording
collect-dagger --policy_type custom --hold_to_intervene true uses the same
messages and scheduler. One control loop sends both the policy’s and the
operator’s commands.
- Holding either grip cancels pending policy work and gives the operator that arm. The other arm holds.
- If the operator’s link drops while grips are held, the operator keeps control and the arm holds its last command.
- With
shadow_inference=true, observations and inference continue during operator control, but the robot discards the replies. - Releasing both grips starts a new generation. The robot resets the endpoint after any outstanding reply, and holds the operator’s last command until a fresh plan arrives. A joint-space handover after IK starts from that held command, with limited velocity and acceleration.
- Grabbing a grip again cancels that handover too. A late prediction from before an intervention can never take control back.
record_joint_actions=truerecords the resolved joint commands, whatever the policy’s action layout.home_on_start=truehomes the arms before the first episode.start_from_current_pose=truekeeps the pose set during scene reset for later episodes.- Pressing
qin the terminal, idle or mid-episode, parks the arms through rest and zero before it turns off the motors.
