collect-data.
For local policies, the VR grip (“side”) buttons drive the intervention during an episode: while the policy is driving, either grip alone freezes it (the robot holds pose and recording pauses); both grips hand control to the operator (teleop engages at the robot’s current pose and recording resumes); either grip alone again hands it back to the policy. Nothing is recorded during the frozen gap, so the dataset flows straight from the policy’s motion into the correction. Remote policies use hold-to-intervene controls. Both modes tag operator rows with LeRobot’s per-frame intervention flag.
Episode boundaries work like collect-data: press the record button to start an episode (the policy starts driving and recording begins), press it again to save, or press it with the reset button held to discard and re-record. On stdin, s saves, r discards, and q discards and quits; --episode_time_s is a safety cap that saves the episode if nothing is pressed. For local policies with --subtasks supplied, typing a number 1..N switches the policy’s instruction mid-episode without ending it.
This command is configured via draccus. Local inference uses --policy_path, --policy_type and --device and runs synchronously in-process. --policy_type custom selects the custom policy interface, with observation acquisition and inference outside the control loop. The recording side mirrors collect-data: nested robot_config / teleop_config settings select cameras, per-joint gains, IK and the VR server. Nest into them with dots or pass a whole-config file. See Command configuration for the syntax and DAgger Collection Config Parameters for the settings reference.
Policy observations use the same exposure-time synchronization as run-policy: the median camera exposure selects the nearest 240 Hz robot-state sample. Local inference publishes its action with that observation’s exposure timestamp. Remote collection instead publishes measured state and the locally dispatched action at each control tick; the recorder pairs camera exposures with this history independently of inference timing. A delayed remote reply therefore does not assign its old observation timestamp to a newly dispatched command.
Local-policy intervention flow
The grip buttons move control between the policy and the operator during an episode. A takeover engages teleop at the robot’s measured pose (the IK worker and smoothing filters are synced to it first, so engaging never drags the arm toward a stale target), and the grippers adopt the controller triggers immediately — pre-set the triggers before you squeeze the grips (hold a trigger down so a held part stays gripped). When you hand back, the policy backend is reset so it re-plans from the corrected pose instead of continuing from its pre-freeze state.
Between episodes the grips work as plain teleop (both to engage, one to disengage — nothing is recorded) so you can reset the scene with the arms, and the VR reset button homes them on demand. The VR reset button and the startup trajectory are disabled inside an episode: episodes end from the record button, and homing is the collision-aware
IKResetController’s job. If the headset link goes stale mid-intervention, tracking auto-disengages and control hands back to the policy.
Remote policies through the custom policy interface
--policy_type custom runs inference on a desktop using
the custom policy interface. It requires --hold_to_intervene true: hold
either grip to control that arm; the other arm holds its pose. Release both
to return to the policy. A lost headset connection with grips held keeps
human ownership and holds position until an explicit release arrives.
The freeze/toggle controls described above apply to local policies only.
For example, on the robot host, replace the endpoint address and camera serial:
--task labels the dataset; the robot does not send
model selection or instruction text to the endpoint. The control panel exposes
these fields through the command schema too. Configure DAgger’s own
server_host and hold_to_intervene settings; its shared defaults follow
collect-data, so run-policy’s shared inference settings do not select the
DAgger endpoint.
The background worker continues requesting predictions during teleop by
default (--shadow_inference false disables this). These predictions never
command motors. Takeover and handback invalidate old requests; handback
resets desktop history and requests a fresh plan from the corrected pose.
The robot holds the last operator command during this wait, then applies
--handover_duration_s (default 0.2667) of joint blending after IK, with a
velocity/acceleration-limited convergence tail. This transition leaves the
published plan and its continuation references unchanged.
Use --record_joint_actions true to record resolved joint commands even
when the policy sends Cartesian targets. The dataset’s declared action
schema and both policy/operator rows then use joints; images and measured
state keep their configured schema. Takeovers keep the same dataset episode
and intervention labels, with a brief capture fence at each source change
so a row cannot pair the previous source’s action with the new source’s label.
--start_from_current_pose true skips the startup home and the mandatory
home after idle teleop. Add --home_on_start true to restore the startup home
while keeping episode starts at the operator’s current pose. The guarded
startup move must finish before the first VR Record gate opens.
Cartesian IK, when selected, is warmed before motors are enabled so its first
solve cannot stall policy dispatch.
--episode_time_s 0 removes the episode time cap.
Instruction selection is managed by the desktop; remote-mode --subtasks
is rejected.
For remote policies, press terminal q once to park through rest and zero
before disabling motors, either at the idle VR Record gate or during an
episode. It needs no Enter and discards any in-flight episode. s + Enter
and r + Enter retain their save/discard behavior during episodes. An idle
q never starts policy execution. EOF, Ctrl-C, faults, declining a contact
retry and failed parking preserve existing motor support when disconnecting;
an already limp contact state remains limp, without a new parking move.
Network teardown follows robot cleanup so a slow peer cannot delay it.
DAgger annotation
Every datasetcollect-dagger creates declares LeRobot’s native per-frame intervention feature — a bool tagged True on rows recorded while the operator was driving and False on the policy’s own rows — the same annotation LeRobot’s DAgger rollout strategy uses, so downstream training can weight or filter the corrections. Local collection omits its frozen gap; remote collection briefly fences capture at ownership changes. Because LeRobot timestamps are index-based, capture re-anchors on resume while retaining one dataset episode. Resuming an existing dataset requires it to already carry the intervention feature (one recorded by this command); a dataset created before the feature existed is rejected rather than recorded into untagged.
The camera and recording plumbing is
collect-data’s out-of-process split, with one twist: the video relay owns the ZED cameras and streams the headset view, and its raw branch is forced onto the shared-memory (pyshm) transport so the frames are readable both here (the policy builds its observations from them) and by the dataset-recorder subprocess. The relay is required — there is no in-process camera fallback, since per-frame camera work in the control process would starve the policy and teleop loops. Install the GStreamer stack (gst.install + gst.build-zed) if the relay can’t start.