VR Interface

The VR interface is a WebXR app that runs in the Meta Quest browser. It sends your controller poses to Axol during teleoperation and data collection.

Connect the headset

You can open the app from two places:

	URL	When to use
Hosted	axol.almond.bot	Most setups.
Self-hosted	`https://<host>:8001/`	Served by `axol serve` on the robot’s computer. Use it offline or when you change the app.

Either way, the headset connects to port 8000 on the robot’s computer. That port is open while axol teleop, axol collect-data, or a teleop or data collection operation from the control panel is running.

Open the app in the Quest browser.
Enter the robot computer’s address in Axol host address. Tap the field to pick one of the last five hosts you connected to.
Press Connect.
The first time, the connection fails because the certificate is self-signed. Tap Authorize certificate, accept the warning in the popup, and the app reconnects. You can also open https://<host>:8000 in the headset browser and accept the warning there.
Press Enter VR.

If Wi-Fi adds lag, tick Quest over USB before you connect. Controller poses then travel over a USB cable, and the cameras keep using the network. See Quest over USB.

Camera views

When camera video is on, the robot’s cameras appear as screens over passthrough. Camera video is on with axol teleop --cameras, with axol collect-data, or when cameras are assigned in the control panel’s Cameras tab.

The overhead camera is in the center. A stereo ZED X overhead camera shows in 3D.
The left and right wrist cameras are in the bottom corners.
Only the cameras being streamed appear.
Connecting cameras… shows until every streamed camera has sent its first frame.

The screens stay fixed in the room, where you were looking when the session started. While arm tracking is disengaged, you can arrange them:

To	Do this
Move a screen	Point a controller at it, hold the trigger, and move the controller.
Resize a screen	Hold both triggers on the same screen, then move your hands apart or together.
Reset all screens	Press B. The screens move to where you are looking and go back to their default size.

Controller bindings

#	Button	Action
1	Left grip	Press both grips together to engage tracking. Once engaged, click a grip to freeze or resume its arm.
2	Right grip	Same as the left grip.
3	Left trigger	Close the left gripper. With tracking disengaged, move or resize a camera screen.
4	Right trigger	Close the right gripper. With tracking disengaged, move or resize a camera screen.
5	X	Return the arms to the rest pose. Cancels a recording countdown. While recording, asks to discard the episode.
6	A	Data collection only. Starts recording, or while recording, asks to save the episode.
7	Y	Exit VR. The arms stop and hold their position.
	B (right)	Reset the camera screens.
	Left thumbstick	Drive Jelly forward, back and sideways.
	Right thumbstick, left or right	Turn Jelly.
	Left thumbstick click, held	Lower Jelly’s lift.
	Right thumbstick click, held	Raise Jelly’s lift.
	Both thumbstick clicks together	Turn box mode on or off. Parcel gripper only.

Engage and freeze the arms

Press both grips together to start tracking.
Click one grip to freeze that arm. It holds its pose and its grip, so it can steady an object while the other arm works. Click the grip again to resume.
A reset, a contact stop or a dropped link disengages both arms. Press both grips to engage again.

With Hold grips to engage turned on in the control panel (--teleop.hold_to_engage true), each grip is a dead-man switch. Hold a grip to move its arm, and release it to freeze the arm. Either grip engages its own arm, including from rest. After a forced disengage, release a grip you are still holding before it engages again.

If the link drops

The arms disengage after about half a second and hold their position if the headset stops sending poses. This happens when you take the headset off, open the system menu, exit the app or lose Wi-Fi. Engaging again never jerks the arms toward the controllers (see Re-engaging). Press X to return them to rest.

Teleop or data collection

The command that starts the session sets the mode. You can’t switch it in the headset.

axol teleop: the HUD shows Teleop and A does nothing.
axol collect-data: the HUD shows Data collection and A records. See Record episodes.

Re-engaging

When a grip engages an arm again, your controller and the arm are usually in different places. This happens after you freeze an arm, walk away, or someone moves the arm by hand. The Ramp button on the HUD chooses what happens:

Button	What happens	Use it when
Ramp: OFF (clutch, the default)	The arm stays where it is. Your controller’s current pose becomes the new starting point.	The arm is holding something, or you want to continue from where it is. Move your hand to roughly match the arm first, so your reach feels the same.
Ramp: ON (ramp)	The arm moves to where your controller is, using the hand-to-arm offset from its last engage. Then it follows your hand.	After a hand-off or after someone guides the arm by hand. Hold your hands where you want the arms and press the grips.

In ramp mode the arm moves at about 0.15 m/s and takes at least 0.75 s. You can keep moving while it catches up. After a reset or a return to rest, the next engage behaves like clutch, so the arms never jump out of rest. Set the startup default with the control panel’s Re-engage setting or --teleop.reengage. During a session, use the Ramp button or Re-engage in session settings.

Box mode

Box mode needs the parcel gripper. Set Robot → Gripper to parcel in the control panel, or pass --axol.gripper parcel. With the standard parallel gripper, box mode and its settings are hidden in the headset and the control panel.

Box mode carries a box with both arms. The grippers press on opposite sides of the box, like two flat hands, and hold it by friction. One controller moves both arms together as a pair. The pair follows your hand’s position and its turn about the vertical axis, and stays level whatever the hand’s pitch and roll.

Turn box mode on

Use any of these:

The Box button on the HUD.
Click both thumbsticks together.
Box mode in session settings.

To start every session in box mode, turn on Box mode at startup in the control panel (--teleop.box_mode true). The pair engages as soon as box mode turns on, led by your right hand. Set Box lead hand to left to lead with the left hand. If someone turns box mode on from the control panel while nobody is wearing the headset, the pair waits for a grip. On engage, the arms first move into a parallel pair over about 1.5 s. They keep their current midpoint and width. Then the leading hand moves the pair.

Bring the grippers roughly into the grasp with normal two-arm teleop before you switch. The button turns blue and reads Box: OFF (aligned) when the grippers face each other across a box-sized gap. Switching then barely moves the arms.

Box mode controls

Control	Action
Leading controller	Move it to move the pair. Turn it about the vertical axis to turn the pair about its center. Pitch and roll do nothing.
Leading grip	Freeze the pair. Press either grip to lead again.
Other grip	Hand the lead to that controller.
Left trigger	Close the left gripper, whichever hand leads.
Right trigger	Close the right gripper, whichever hand leads.
Thumbstick left or right	Change the grip width. Right is wider, left is narrower.
Thumbstick forward or back	Move the elbows out (forward) or in (back). See Elbows.
Thumbstick click	Switch between the straight and flush grasp. See Grasps.
Both thumbstick clicks	Leave box mode.
X	Return to rest and leave box mode.

Both thumbsticks do the same thing. They shape the grasp but never move the pair. A push counts only along its main direction, so widening the grip never moves the elbows. To clamp a box, lead the pair to it. Then push a thumbstick left until the gripper faces close on the box’s sides. The width is the gap between the two contact faces, from 0.02 m to 0.70 m.

Grasps

Grasp	How the grippers sit
`straight` (default)	Fingers point straight forward. The parcel gripper’s folded plate lies flat on the box.
`flush`	Each gripper turns 39° inward about the front edge of its plate. The plate’s chamfer and the tip of the fixed blade both touch the box. Release the triggers before you close the width.

Click either thumbstick while leading to switch. The arms move to the other grasp over about 1.5 s, and you can switch with a box already clamped. The Box button shows the current grasp, for example Box: ON (straight). Set the flush angle with Angled yaw and the starting grasp with --teleop.box_grasp.

The parcel gripper

The parcel gripper has a fixed blade and a hinged plate. When you release the trigger, the plate opens fully and folds back along the wrist, on the side facing the box. Box mode places each gripper by this contact face, not by its mount. Each gripper turns whichever flat side is closer toward the box, so the wrist never makes a half turn. If a gripper engages with its fixed blade toward the box, or with its motor cap facing down, set Left face or Right face to +x or -x. Try the other value if the first is wrong, then engage again. If the pair doesn’t sit flat on the box, adjust Angled yaw or --teleop.box_grip_tilt.

Driving with the box

While a grip leads the pair, the thumbsticks shape the grasp and Jelly stays still. Freeze the pair to drive Jelly with the normal controls.

Clamp the box.
Click the leading grip to freeze the pair. The arms hold the box where it is, with the grippers closed.
Let go of both thumbsticks. The sticks start driving Jelly once both are released.
Drive to where you need to go.
Squeeze both triggers, then press a grip to lead the pair again.

When a grip takes the lead, each gripper follows its trigger again. If the triggers are released at that moment, the grippers open and the box drops.

Elbows

Push a thumbstick forward or back to set Elbows out. It goes from 0° (hanging below the line from shoulder to wrist) to 90° (level), and starts at 30°. Bring the elbows in to pass a doorway, or out for a wide box. The value is kept for the next engage. If closing the width would bring an arm too close to the torso, the width stops closing. To turn off elbow control, pass --teleop.box_elbow_weight 0. The elbows then rest where they fall, and forward or back on the stick does nothing.

Clamp force

The clamp gets harder the further you push the width past the box, by about 6 N per arm for each centimetre. These settings shape the squeeze. They work on the real robot only.

Setting	Default	What it does
Squeeze force	0 (off)	Caps the clamp force per arm, in newtons. Try 8 N for a light parcel, and raise it if boxes slip.
Squeeze lean	1×	Tilts each gripper into the box as you squeeze, so the blade tip and the plate press evenly. Raise it if the tip lifts, lower it if the plate lifts. 0 turns it off.
Squeeze trim	5°	The most extra inward turn added while pressing, so each gripper face ends up flat on the box. 0 turns it off.
Squeeze cap	0 (off)	Caps the shoulder torque that squeezes the box. Leave it off unless you need it, because it also slows an arm moving toward the other and skews the pair.

The wrists also limit the clamp to about 31 N per arm in a typical carrying pose. To clamp harder, raise --teleop.box_wrist_torque (default 5 Nm). The wrist motors then run hotter. While the grippers press, the Box button shows the clamp force and trim, for example Box: ON (straight, 6 N, 2° in). The teleop log prints a box clamp: line every second.

Leave box mode

Click both thumbsticks together, press the Box button, or turn off Box mode in the control panel. Press both clicks within about a third of a second of each other. A quick double click counts as one toggle. Both arms engage in normal teleop as box mode turns off:

With Ramp: ON, each arm moves back to where its controller is, using the hand-to-arm offset from before box mode.
With Ramp: OFF, the arms stay where they are and follow your hands from there.

If you leave while leading in the flush grasp, the pair first returns to straight over about 1.5 s. To leave right away, release the grip or toggle again. Pressing X always leaves box mode and returns the arms to rest. After a hand-off, or after someone moves the arms by hand, press a grip. Box mode always starts by moving the arms into the parallel pair at their current midpoint and width.

Session settings

You can change these settings during a session. Open them with the Settings button on the HUD, or in the Session settings card of the control panel. Both show the same values, so a change in one appears in the other. Box and Ramp also have their own HUD buttons.

Setting	What it does
Box mode	One controller drives both arms as a pair. See Box mode.
Re-engage	`clutch` or `ramp`. See Re-engaging.
Hold to engage	Makes each grip a dead-man switch instead of a toggle.
Elbows out (°)	Box mode. How far out the elbows are held, from 0 to 90.
Squeeze lean (×)	Box mode. How far each gripper tilts into the box as you squeeze. Real robot only.
Squeeze trim (°)	Box mode. The most extra inward turn added while pressing. Real robot only.
Box lead hand	Which controller leads the pair when box mode turns on.
Box grasp	`straight` or `flush`.
Angled yaw (°)	How far the `flush` grasp turns each gripper inward. Default 39.
Left face / Right face	`auto`, `+x` or `-x`. Which side of each gripper faces the box.
Squeeze force (N)	Box mode. Caps the clamp force per arm. 0 means no cap. Real robot only.
Squeeze cap (Nm)	Box mode. Caps the shoulder torque that squeezes the box. 0 is off. Real robot only.
Grip force (Nm)	How hard a closed gripper squeezes, on both hands. Real robot only.
Reach scale (×)	Hand motion multiplier. Above 1 extends your reach, below 1 gives finer control.
Arm speed (rev/s)	Joint speed limit while tracking. Lower it for slow, careful work.

The box mode settings appear only with the parcel gripper. Sessions start with the values from the command line or the control panel’s saved settings. Changes made here last for the current session only.

Jelly

On Axol on Jelly, the thumbsticks drive the base, so you can move Axol without leaving VR. Jelly is a powered base with omni wheels and a telescoping lift.

Control	Action
Left thumbstick	Drive forward, back and sideways.
Right thumbstick, left or right	Turn.
Left thumbstick click, held	Lower the lift.
Right thumbstick click, held	Raise the lift.

Jelly moves only while you push a stick, and stops smoothly when you let go. This works whether or not arm tracking is engaged.
In box mode, the sticks drive Jelly only while the pair is frozen. See Driving with the box.
If the headset link drops while you drive, Jelly stops. A parked Jelly keeps holding its position.
Jelly’s motion is never recorded in a dataset, and a policy never drives it. You can move Axol during teleop or data collection without affecting the recording.

Set up Jelly

Axol drives Jelly when Jelly’s CAN adapters are plugged in. The wheels use can_alm_axol_b. The lift uses can_alm_axol_c, or shares the wheel bus. The control panel names the adapters when they first appear (CAN discovery), or you can run axol can.setup. To leave attached hardware out of a session, use the switches in the Robot tab of Axol settings:

Switch	Flag
Axol arms	`--arms false`
Jelly wheels	`--jelly.wheels false`
Jelly lift	`--jelly.lift false`

With Axol arms off, the session drives only Jelly and never connects to the arms. The same happens when Jelly is attached but the arm hub is not. Jelly’s other settings are under Advanced → Jelly, or the --jelly.* flags of axol teleop. Home the lift once with axol lift.home before it can move. axol lift.goto raises it to the install height. Both are also in the diagnostics dashboard.

Record episodes

In data collection, record with A and X:

Press A. Recording starts after a 3-second countdown. Press X during the countdown to cancel.
To finish, press A. Save episode? appears. Press A again to save, or X to keep recording.
To throw the take away, press X. Discard episode? appears. Press X again to discard it and record again, or A to keep recording.

Nothing is saved until you confirm. While an episode is being saved, every control except Y is blocked. The HUD shows Episode N at the top right, the number of the episode you are about to record. If the robot reports an error, the headset shows an error indicator. For the messages behind these states, see the VR server API.

Run the app yourself

The app’s source is in the web/ folder of this repository. The same build produces the control panel. From web/:

npm install

# Build the client package first, then the app (output: web/app/dist, served by axol serve)
npm run build --workspace=packages/axol-vr-client
npm run build --workspace=app

# Development server with hot reload
npm run dev --workspace=app

For development, open the printed URL in the Quest browser and connect as usual.

Get Started

Operations

Mantis

Remote Teleop

Web Interfaces

Advanced

VR Interface

Connect the headset

Camera views

Controller bindings

Engage and freeze the arms

If the link drops

Teleop or data collection

Re-engaging

Box mode

Turn box mode on

Box mode controls

Grasps

The parcel gripper

Driving with the box

Elbows

Clamp force

Leave box mode

Session settings

Jelly

Set up Jelly

Record episodes

Run the app yourself

Next steps

Teleoperation

VR server API

​Connect the headset

​Camera views

​Controller bindings

​Engage and freeze the arms

​If the link drops

​Teleop or data collection

​Re-engaging

​Box mode

​Turn box mode on

​Box mode controls

​Grasps

​The parcel gripper

​Driving with the box

​Elbows

​Clamp force

​Leave box mode

​Session settings

​Jelly

​Set up Jelly

​Record episodes

​Run the app yourself

​Next steps

Teleoperation

VR server API

Connect the headset

Camera views

Controller bindings

Engage and freeze the arms

If the link drops

Teleop or data collection

Re-engaging

Box mode

Turn box mode on

Box mode controls

Grasps

The parcel gripper

Driving with the box

Elbows

Clamp force

Leave box mode

Session settings

Jelly

Set up Jelly

Record episodes

Run the app yourself

Next steps