Secure WebSocket server (WSS) that receives VRFrame JSON messages from the VR app. A self-signed TLS certificate is auto-generated in ~/.almond/vr/certs/ on first use. This module can be used standalone to read raw VR data without full teleoperation — useful for custom control loops or data collection.
Or use a callback instead of polling:
VRFrame fields
Pose transports & USB/Wi-Fi fallback
The headset can stream pose over more than one transport at once — the network WebSocket, a USB adb reverse tunnel (Quest over USB), and a dedicated WebRTC data channel (remote teleop). Each frame carries a shared seq; the server keeps a monotonic high-water mark and processes each seq exactly once, from whichever link delivers it first — so on_frame fires once per logical frame, a quiet USB link fails over to Wi-Fi instantly, and a resuming link can’t rewind to a stale pose. The high-water mark resets when the last client disconnects and on disable(), so a reloaded headset (its counter restarted) isn’t gated against a stale value.
The WebRTC pose data channel (almond_axol.vr.control_channel) shares the same ICE/TURN path as the camera video, so an operator off your tailnet can drive over a relay. ICE servers are read from the environment (AXOL_TURN_URL / AXOL_TURN_USERNAME / AXOL_TURN_PASSWORD; see almond_axol.vr.ice) — on a LAN or plain tailnet they’re unset and the direct path is used. If the data channel fails to negotiate (or the peer connection drops), the headset re-requests it after a backoff rather than permanently falling back to the WebSocket.
Server → headset feedback
The server can push a state override to all connected headsets at any time using VRServer.broadcast_text(). The headset interprets messages of the form {"type": "state", "value": "saving"} as a state override that blocks recording controls. {"type": "state", "value": "error"} shows an error indicator in the headset UI. {"type": "state", "value": "data_collection"} re-enables controls after saving. The AxolVRTeleop.send_feedback_state(state) helper wraps this for all VRState values.
On connect the server also announces its operating mode — {"type": "mode", "value": "teleop" | "data_collection"} — set via VRServer.set_mode(). The web UI uses it to lock the HUD to a single mode: axol teleop sends teleop (recording controls hidden), while axol collect-data sends data_collection (recording enabled, but no switching back to plain teleop). The message is pushed once per client in the WebSocket accept handler, so a headset that joins mid-session still gets it.
During data collection the server likewise pushes the current episode number — {"type": "episode", "value": N} (1-based) — which the headset renders as an Episode N HUD readout. axol collect-data announces it at the start of each episode via AxolVRTeleop.send_feedback_episode(episode), which both stores the value on the server (VRServer.set_episode(), re-sent on connect so a mid-session headset shows the right number) and broadcasts a live update to connected clients. Plain teleop never sets it, so the readout stays hidden.
Teleop additionally broadcasts {"type": "tracking", "value": true|false} whenever the engage toggle changes, so the headset knows when the robot is being controlled — it uses this to only allow repositioning and resizing the camera screens (a trigger gesture) while tracking is disengaged.
Camera video to the headset
The server can stream camera frames to the headset over WebRTC. Register per-camera sources with set_video_sources(). A source is either a pre-encoded camera (the GPU-resident gst_zed pipeline, exposing subscribe() / alive — its H.264 access units go straight to RTP with no Python encode) or a raw frame source. A connected ZedCamera (or stereo eye) is registered directly — the relay adapts it — and you can equally pass any object exposing fixed width / height / fps plus wait_next(after_ts, timeout_ms). The manager picks the right WebRTC track per source automatically:
The existing /ws channel is reused for SDP signaling (no extra ports): the headset sends webrtc-request, the server replies with a webrtc-offer carrying one video track per source (labelled by camera name), and the headset answers with webrtc-answer. A stereo overhead streams both eyes packed side-by-side in one track, overhead_sbs — a single decoder session on the headset, since two full-res sessions cap the Quest’s render rate — which the VR app renders per-lens through a WebXR media layer; the SDK-fallback path (no gst) instead registers the two per-eye sources overhead_left / overhead_right. A stereo wrist is registered as a single source under its slot name (one feed, like a mono camera). Dataset recording is unaffected — it still stores the two per-eye observation keys.
Pass None or {} to turn video off. axol teleop --cameras and collect-data wire this up automatically.
Pipeline
Teleop and data collection share one GPU-resident camera path (almond_axol.video.gst_zed). The zed-gstreamer source elements (zedxonesrc mono / zedsrc stereo) grab frames straight into NVMM (GPU) memory, and a single in-process GStreamer pipeline tees that one zero-copy buffer to two consumers:
- an encoded branch —
nvv4l2h264enc (NVENC) → appsink — so the whole grab→encode chain stays on the GPU (~5–10 ms) and Python only ever sees the H.264 access units, which aiortc forwards as pre-encoded packets (encoder.pack, no Python encode);
- a raw branch —
nvvidconv (the VIC) converts to RGBA and downscales to the dataset resolution — built only when raw frames are needed (the dataset, policy inference). For the dataset it ends in a gst-native shmsink that exports each frame to the recorder subprocess in C (no Python pull in the relay); for in-process consumers like policy inference it ends in an appsink pulled into numpy. Each frame still carries a capture_perf_ts aligned to the joint sample (the in-process path maps the buffer PTS onto time.perf_counter; the dataset path stamps receipt time minus the relay’s measured pipeline latency), so dataset rows align image capture with the joint sample — without the ZED SDK’s ~26 ms host round trip.
Both teleop and data collection run the camera grab/encode/WebRTC pump in a dedicated subprocess (almond_axol.video.video_proc.VideoRelayProcess, registered via set_video_manager()): isolating it keeps the control loops at full rate (running it in-process measurably halves the IK rate). For teleop the subprocess opens the cameras with the raw branch off (encoded only). For data collection it opens both branches — the encoded branch streams the headset view, and the raw branch is exported to a second dedicated recorder subprocess (almond_axol.recording.record_proc.DatasetRecorderProcess) that owns the LeRobotDataset end to end. On the Jetson that export is a gst-native shmsink→shmsrc transport, so the relay runs no Python per raw frame — keeping its single-threaded aiortc send uncontended during recording — with a multiprocessing shared-memory copy (almond_axol.video.shm_frames, RawFrameReader) as the fallback when gst’s shm plugin is absent. The control loop only writes a small joint/action snapshot to shared memory each tick (SnapshotWriter) and never touches the grab/encode/dataset path; it still receives each camera’s dimensions via AxolRobot.set_external_cameras() to size the dataset features. Capture timestamps stay aligned because time.perf_counter is CLOCK_MONOTONIC (a shared origin across processes), so a capture_perf_ts stamped in the relay is directly comparable to the joint-sample timestamps. aiortc handles the WebRTC transport (RTP/SRTP/ICE) and SDP signaling in both cases. When the gst stack is unavailable, data collection falls back to the in-process camera path (set_video_sources() with the robot owning the cameras), and both paths fall back to the ZED Python SDK grab (ZedCamera) plus in-Python NVENC (almond_axol.video.hw_video).
Requires the GPU-resident camera stack the host installer sets up once: the GStreamer + PyGObject support from axol gst.install plus the patched zed-gstreamer source plugins from axol gst.build-zed (the NVENC elements themselves ship with the Jetson L4T BSP). The NVENC/VIC engine clocks must also be pinned to their maximum for low-latency encode (axol jetson.setup, run at boot by the installer’s systemd unit); the default Jetson devfreq governor scales them for throughput, which otherwise roughly triples per-frame encode latency.
VRServerConfig fields
Before opening the VR app, accept the self-signed certificate by navigating to https://<hostname>.local:8000 in the VR browser and proceeding past the security warning.
