Data Collection

Data collection records teleoperation episodes and camera video into a LeRobot dataset you can train on. Start it from the web control panel or with axol collect-data.

Axol records joint positions by default. Turn on Observe Cartesian to record gripper poses instead.
Mantis records gripper poses from Quest, Lighthouse or Ultimate tracking.

Run it on the computer that is wired to the hardware and has the ZED cameras attached, usually a ZED Box or Jetson. Images and actions then share one clock.

Before you start

Install Axol with the one-command install and connect the device you selected.
- For Axol, connect the robot.
- For Mantis, finish its hardware and pose source setup. Mantis settings → Tracking must show setup complete, with an approved tracker-to-gripper transform for both sides.
Connect at least one ZED camera over GMSL. Assign it to a slot with Record turned on.
- Axol uses the overhead, left arm and right arm slots.
- Mantis uses separate Head, Left Mantis and Right Mantis slots.
- Both can record the stereo head camera and both wrist cameras together.
For Axol, and for Mantis with the Quest source, have a Meta Quest with the TLS certificate authorized. Mantis with Lighthouse or Ultimate needs no headset.

To list the connected cameras and their serial numbers, run:

python -c "from almond_axol.zed import list_zed_devices; print(list_zed_devices())"

If a camera plugged in after boot doesn’t show up, press Restart camera service under Axol settings → Cameras or Mantis settings → Cameras. On the CLI, run sudo systemctl restart zed_x_daemon.

Run it

Axol
Mantis

Connect the robot and assign cameras

Connect the Axol Host and, with Axol selected, the robot. Open Axol settings → Cameras and assign each camera to the overhead, left arm or right arm slot. Turn on Record for at least one. Axol detects stereo ZED X cameras automatically. See Cameras.

Select Collect data and fill in the fields

Pick Collect data and fill in:

Repo ID: type a new name such as myorg/pick-place, or pick an existing dataset from the list to add to it. The list shows each dataset’s episode count.
Task: a description of the task.

Recording rate, teleoperation rate, stiffness and upload settings are in the Settings tabs.

Start and record from the headset

Press Start, connect the headset and drive the arms. Record with the controllers, as described in Recording an episode. The panel shows the camera feeds and episode state, and has buttons for the same actions. See Driving from the panel.

From the CLI, on the robot’s computer:

axol collect-data \
    --repo_id myorg/pick-place \
    --task "Pick the red cube and place it in the bin" \
    --robot_config.cameras "{overhead: {serial: 41234567}, left_arm: {serial: 41234568}, right_arm: {serial: 41234569}}"

Replace the serial numbers with your cameras’. Assign at least one slot. Slots you leave out aren’t recorded, so --robot_config.cameras "{overhead: {serial: 41234567}}" records the overhead camera only.The cameras value is one inline YAML dictionary. See Command configuration.Axol detects stereo ZED X cameras from their serial number:

A stereo overhead camera records both eyes, as overhead_left and overhead_right.
A stereo wrist camera records its left eye under the slot name, like a mono camera.

To override the detection, set stereo: true and choose the eyes:

eyes: both records X_left and X_right.
eyes: left or eyes: right records one eye under the slot name X. A forced stereo wrist camera needs eyes: left to stay single-eye.

For example, to force a stereo overhead camera:

    --robot_config.cameras "{overhead: {serial: 41234567, stereo: true}, left_arm: {serial: 41234568}, right_arm: {serial: 41234569}}"

Connect the headset and record from the controllers. See Recording an episode. Press Ctrl+C to finish. See collect-data for every field, including --fps, --teleop_hz, stiffness and --push_to_hub.

Before you record training data, pass the Mantis tracking acceptance with the same pose source. Collection refuses to start with a missing or unapproved tracker-to-gripper transform. The built-in VIVE factory transforms count as approved.

Select Mantis and check its settings

Press Select on the Mantis tile and let it connect. Click the tile’s title to open Mantis settings, then check:

Tracking: the pose source’s setup is complete. See Mantis Tracking.
CAN: the two channels are different.
Triggers: the trigger endpoints are checked.
Cameras: at least one Head, Left Mantis or Right Mantis camera has Record on. For stereo head video, turn on both L and R under Record.

Select Collect data and fill in the fields

Pick Collect data and fill in Repo ID and Task as for Axol. Start stays unavailable until the pose source, both transforms, the Mantis link and the cameras are ready. The Before you can start list names what is missing.mantis_allow_uncalibrated is only for a calibration capture that won’t be used for training.

Start and bring up the pose source

Press Start.

Quest: open axol.almond.bot and enter VR. Left and right come from the controllers.
Lighthouse or Ultimate: power on both trackers inside the tracked area. Wait until both trackers report a pose and both triggers report in. No headset is needed.

Record with the source's controls

Quest
Lighthouse / Ultimate

Hold the rigs at the start pose and release both controller grips. Press both grips to align and engage. Then record with A and X as on Axol (see Recording an episode), or with the panel’s episode buttons.

Re-record after any tracking break

If a tracker loses its pose, a trigger stops reporting or tracking disengages, the rig freezes and the take fails its quality check. The panel shows why the take wasn’t saved. See Driving from the panel.After every take, tracking disengages and the grippers turn off. To start the next take, restore both sides if needed, then repeat the start gesture wherever the rigs are. Nothing resumes automatically.

From the CLI, collect-data starts the tracker bridge and trigger readers itself. Without --mantis_source, it uses the source saved in the settings.

axol collect-data --mantis true --mantis_source lighthouse \
  --repo_id myorg/mantis-pick --task "Pick the part" \
  --robot_config.cameras "{overhead: {serial: 41234567}, left_arm: {serial: 41234568}, right_arm: {serial: 41234569}}"

The overhead slot is the head camera. It records both stereo eyes as overhead_left and overhead_right. Leave out any slots you don’t use.Set the source to quest or ultimate as needed. Don’t start a second tracker.bridge. See collect-data and the tracker reference.

Quest controller layout

With Axol, and with Mantis on the Quest source, you record from the Quest controllers. The arm controls are the same as in teleoperation. Data collection adds the A button for recording. The control panel can take the same actions. See Driving from the panel.

#	Button	Action
1	Left grip	Press both grips together to engage. After that, each grip freezes or resumes the arm on its side. With Hold grips to engage on, hold a grip to move that arm.
2	Right grip	Same as the left grip.
3	Left trigger	Close the left gripper. While disengaged, point at a camera screen and hold to move it. Use both triggers on one screen to resize it.
4	Right trigger	Close the right gripper. While disengaged, moves and resizes camera screens like the left trigger.
5	X	Reset the arms to the rest pose, including out of a contact hold. Cancels the countdown. During a take, asks Discard episode?
6	A	Record: start a take after a 3-second countdown. During a take, asks Save episode?
7	Y	Exit the VR session.
	Right B	Reset the camera screens to where you are looking. See Camera views.
	Thumbsticks	Drive Jelly between takes, on Axol on Jelly. Jelly motion is never recorded.

The camera feeds are shown in the headset, so you can line up takes from the robot’s view.

Recording an episode

Drive to the start pose

Put on the headset, engage tracking and drive the arms to where the take should begin.

Start a take

Press A. Recording starts after a 3-second countdown. Press A again or X to cancel the countdown.

End the take

The first press asks for confirmation in the headset, so a stray press never loses a take.

Press A for Save episode? Press A again to save, or X to keep recording.
Press X for Discard episode? Press X again to discard and re-record, or A to keep recording.

After you confirm a save, the headset shows Saving and only Y works until the episode is written.

Let the arms return to rest

After each take, the arms return to the rest pose on their own. The move is guarded: if an arm meets resistance, such as a gripper caught on a handle or a person, the move stops.

The arms go limp in a gravity-compensation hold. The episode keeps saving.
Guide the arms clear by hand.
Press X to plan a new return from where they are.

You can’t record while the arms are limp. Pressing A reminds you to reset first.The grips don’t engage tracking until the arms are home and the episode is saved or discarded. Then release both grips and squeeze both again. A grip held through the save doesn’t count.

The HUD shows Episode N at the top right: the number of the episode you are about to record, counting from 1.

Headset state	Meaning
`Data collection`	Driving the arms, ready to record
`Recording`	Recording frames to the dataset
`Saving`	Writing the episode to disk. Controls are blocked.

If --root already holds the dataset, collection adds to it. If you stop a new session before saving any episode, Axol deletes the empty dataset.

Contact stops

Every return to rest is guarded, including the X reset and the first move at startup. Set the sensitivity with Return-to-rest contact stop in Axol settings → Robot (--teleop_config.vr_teleop_config.reset_torque_threshold). Raise it if normal returns stop for no reason. Set 0 to turn it off. You can also stop the arms when they push hard during a take. This check is off by default, because recording often pushes on the scene on purpose. Set Teleop contact stop in Axol settings → Robot (--teleop_config.vr_teleop_config.teleop_torque_threshold). The panel suggests 16 Nm. When it trips, tracking disengages, the take is discarded and the arms go limp. Guide them clear, then press X and record again. Teleoperation uses the same settings. These stops apply to Axol’s arms. Mantis has no arms to return home. Each Mantis take starts with the source’s start gesture, and the grippers are off between takes.

Driving from the panel

When you start from the control panel, it also works as an operator display. It shows the camera feeds the headset receives, with a full-screen option. The Episode control box shows the phase, the episode number and the saved count, with these buttons:

Start recording starts the same 3-second countdown as the headset. Click it again to cancel.
Save episode and Discard episode ask you to click again to confirm.
Return to rest appears during a contact hold. It does the same as X.

The controllers keep working. Whichever input arrives first ends the take. Anything you start on the controllers, such as a confirmation or a countdown, also shows in the panel. See Episode controls. When a take ends without being saved, the Episode control box shows the reason in red, and the log shows it too. The message stays until the next take starts recording. A take isn’t saved when:

the Mantis quality check rejects it,
a camera fails during capture,
it ends before its first frame is recorded, or
the recorder refuses it at save.

Saved data

Each episode is on disk as soon as it is saved. If the recorder stops unexpectedly, you lose at most the take being written.
Resuming checks the dataset. If the saved episodes have a gap, Axol keeps the complete episodes before the gap, removes the rest and continues from there. If no complete episode is left, it asks you to delete the dataset.
Takes with dropped video frames are discarded. If the camera encoder drops frames under heavy load, the video doesn’t match the recorded rows. Axol discards that take and logs why. The session keeps running, so record the take again.

See the VR Interface guide for every controller binding and headset state.

Mix Axol and Mantis episodes

Axol and Mantis episodes can go in one dataset. For each Axol run:

Turn on Observe Cartesian, so the recorded state and actions match Mantis.
Keep the gripper setting, camera slots, image resolution and recording rate the same as the Mantis runs.

A dataset started on Mantis may have an observation.pose_lag column, which Axol fills in automatically. A dataset started on Axol without that column keeps its layout, and Mantis leaves the column out.

Next steps

Config Parameters

Every setting for data collection, with its default.

Run Policy

Run a trained policy on the same computer.

collect-data reference

Every flag for data collection.

Get Started

Operations

Mantis

Remote Teleop

Web Interfaces

Advanced

Data Collection

Before you start

Run it

Quest controller layout

Recording an episode

Contact stops

Driving from the panel

Saved data

Mix Axol and Mantis episodes

Next steps

Config Parameters

Run Policy

collect-data reference

​Before you start

​Run it

​Quest controller layout

​Recording an episode

​Contact stops

​Driving from the panel

​Saved data

​Mix Axol and Mantis episodes

​Next steps

Config Parameters

Run Policy

collect-data reference

Before you start

Run it

Quest controller layout

Recording an episode

Contact stops

Driving from the panel

Saved data

Mix Axol and Mantis episodes

Next steps