Skip to main content
Connects a VR headset to Axol, or to the simulator, for teleoperation. Each VR frame goes through four stages before it reaches motion_control():
  1. A lag-compensated low-pass filter on the raw poses.
  2. An IK solve, in a separate process so it never blocks the asyncio event loop.
  3. Exponential smoothing of the IK output.
  4. A trapezoidal velocity and acceleration limit.
Each config you leave as None is loaded from the shared settings file, ~/.almond/settings.json, over the built-in defaults. The control panel edits this file and axol teleop reads it, so an SDK session uses the same rest poses, multipliers, IK weights and VR port. Pass a config to override it. See Shared settings. Use step() instead of run() to run teleoperation inside your own control loop:
To show camera feeds in the headset, call VRTeleop.set_video_sources() inside the session. It forwards the sources to the VR server’s WebRTC relay, which needs the GStreamer NVENC stack from axol gst.install. Each value is a connected ZedCamera, or any object with width, height, fps and wait_next(). See almond_axol.vr.
The teleop CLI command is the command-line equivalent. Its --cameras flag sets up the ZED cameras for you.
set_video_sources() runs the WebRTC relay in the teleop process. VRTeleop.set_video_manager() instead runs camera capture, encoding and WebRTC in a separate process (almond_axol.video.video_proc.VideoRelayProcess), so video work can’t slow the control loops. The CLI uses set_video_manager(). Both send the same video to the headset.

VRTeleopConfig fields

Joint velocities and accelerations are set in rad/s and rad/s². The defaults below also give them in rev/s, the unit the control panel and headset show.

Control loop and filtering

record writes each pipeline stage to its own file: <prefix>_ik.npz, <prefix>_cmd.npz, <prefix>_meas.npz and <prefix>_rt.npz. The rt file holds the realtime core’s 240 Hz motor commands, control terms, timing and feedback. Each file keeps the most recent engage-to-disengage segment.

Engage and reset

Box mode

Box mode needs the parcel gripper. Set the gripper on the robot with --axol.gripper, not with this config. A gripper value that disagrees with the robot’s is overridden with a warning, and with no robot to follow (Sim), None means "parallel". With the parallel gripper:
  • every box_* live setting is hidden,
  • box_mode starts off, with a warning,
  • VRTeleopCore.set_box_mode(True) raises ValueError.
In box mode one grip drives both arms by position and heading. The pair follows the hand’s movement and its turn about vertical, and stays level whatever the hand’s pitch and roll. While a grip leads, either thumbstick shapes the grasp:
  • Left and right change the grip width. Right is wider.
  • Forward and back move the elbows. Forward is farther apart.
  • A click toggles between the two grasps.
Freeze the pair and the thumbsticks drive Jelly instead. Box mode switches from the headset’s Box button, both thumbstick clicks together, the control panel, or VRTeleopCore.set_box_mode(). The switch engages right away: into box mode led by box_lead_hand, or out of it with both arms. If no VR frame arrives within a second of the switch, or a return to rest ends box mode, nothing engages.

The squeeze

When you jog the grip width past the box, the arms’ joint springs turn the extra distance into clamp force, about 6 N per centimetre. These settings shape the clamp:
  • Squeeze force (box_squeeze_force) caps the clamp force per arm, at any pose. Try 8 N. Whatever it is set to, the clamp stays under about 31 N a side, the most the wrists can press evenly at the default box_wrist_torque.
  • Squeeze lean (box_squeeze_lean) yaws each gripper inward about 1.3° per centimetre of clamp depth, so the tip and root of the gripper press evenly. Raise it if the tip lifts as you squeeze. Lower it if the face near the wrist lifts.
  • Squeeze trim (box_squeeze_trim) adds inward yaw from the measured gripper angle until each face sits parallel to the box. It corrects for backlash and flex that the lean doesn’t model, and acts only once the pair has held still for 0.3 s.
  • Squeeze cap (box_squeeze_torque) is a hard torque cap on the two shoulder joints. It also slows any move toward the other arm, so the pair can skew. Leave it at 0 unless you need it.
The squeeze settings and box_wrist_torque work on hardware only. The IK worker reports the clamp force and trims in IKWorker.pair_status() as squeeze, trim (mean) and trims ([left, right]). While the grippers press, the teleop log prints a box clamp: line once a second.

Mantis

A TCP transform must have seven finite numbers, a translation of at most 1 metre, and a nonzero quaternion. Production Mantis collection also requires it to match the active tracker’s entry in ~/.almond/mantis/tcp_transform.json, or a verified factory constant. Lighthouse and Ultimate trackers take their left and right bindings (survive:<id>, ultimate:<mac>) from the tracker config instead of tracker_key. A saved Ultimate entry stops counting if its pose convention changes. Set urdf_viewer_world_aligned to true for a Lighthouse or Ultimate tracker only after you have registered its origin and heading to the Quest’s world.

Engage behavior

  • From rest, press both grips together to engage both arms.
  • Once engaged, a grip click toggles its arm between tracking and frozen. A frozen arm holds its pose and grasp while the other keeps tracking.
  • With hold_to_engage, an arm tracks only while its grip is held, from rest too. Releasing the grip freezes that arm. If tracking stops while a grip is held, release it once before it engages again.
  • The reset button moves both arms back to the rest pose. The move avoids self-collision when KinematicsConfig.self_collision is on, and otherwise stays within joint limits.
  • In box mode, turning box mode on engages the pair, led by box_lead_hand. From a frozen pair, a single grip engages both arms again, and that controller leads.

Re-engaging

When a grip re-engages an arm after a freeze, a disengage, or someone moving the arm by hand, reengage decides which side moves:
  • "clutch": the arm stays put, and the controller’s current pose becomes the new origin.
  • "ramp": the arm keeps the controller-to-arm mapping from its previous engage and blends to where that mapping puts the controller. The blend is paced by reengage_ramp_speed, takes at least reengage_ramp_min_s, and follows a moving hand.
The arm never moves on the engage frame itself. A return to rest or a DAgger sync drops the mapping (IKWorker.reset(), clear_engage()), so the next engage starts fresh in either mode. Engages inside box mode ignore reengage. Leaving box mode with "ramp" blends both arms back to the mapping from before box mode.

Live settings

Some settings can change while a session runs, from the headset or the control panel: box_mode, the other box-mode settings marked live above, reengage, hold_to_engage, position_multiplier, teleop_max_vel, and the gripper torque limit on hardware. almond_axol.teleop.live.LiveSettings validates each request and passes it to VRTeleopCore.set_live(key, value). Changes apply on the IK thread before the next frame, or while the session is idle, so the control panel can change them with no headset connected. See Live session settings for the full list and the wire protocol.