> ## Documentation Index > Fetch the complete documentation index at: https://docs.almond.bot/llms.txt > Use this file to discover all available pages before exploring further. # almond_axol.vr > Secure WebSocket server that receives VRFrame messages from the VR app. Secure WebSocket server (WSS) that receives `VRFrame` JSON messages from the VR app. A self-signed TLS certificate is auto-generated in `~/.almond/vr/certs/` on first use. This module can be used standalone to read raw VR data without full teleoperation — useful for custom control loops or data collection. ```python theme={null} from almond_axol.vr import VRServer, VRServerConfig ``` ```python theme={null} import asyncio from almond_axol.vr import VRServer, VRServerConfig async def main(): async with VRServer(VRServerConfig(port=8000)) as vr: while True: frame = vr.get_frame() if frame is not None: print(frame.l_ee, frame.r_ee) await asyncio.sleep(0.01) asyncio.run(main()) ``` Or use a callback instead of polling: ```python theme={null} def on_frame(frame): print(frame.l_grip, frame.r_grip) async with VRServer() as vr: vr.set_on_frame(on_frame) await asyncio.sleep(float("inf")) ``` ## `VRFrame` fields | Field | Type | Description | | --------------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `l_ee` / `r_ee` | `VRPose` | 6-DOF end-effector pose (position + quaternion) | | `l_elbow` / `r_elbow` | `VRPosition` | 3D elbow positions | | `l_grip` / `r_grip` | `float [0, 1]` | Gripper commands | | `l_lock` / `r_lock` | `bool` | Grip toggles (engage/disengage tracking) | | `reset` | `bool` | Rising edge triggers a reset move | | `state` | `VRState` | `TELEOP`, `DATA_COLLECTION`, or `RECORDING` (headset-driven); `SAVING` and `ERROR` are server-pushed only | | `t` | `float \| None` | Client capture timestamp in ms (`performance.now()`), used by the pose interpolator. `None` falls back to latest-wins | | `seq` | `int \| None` | Monotonic frame counter set by the headset. The same `seq` arrives over both the USB and network transports; the server processes each logical frame once, via whichever transport delivers it first. `None` disables cross-transport de-duplication | ### Pose transports & USB/Wi-Fi fallback The headset can stream pose over more than one transport at once — the network WebSocket, a USB `adb reverse` tunnel ([Quest over USB](/guides/quest-over-usb)), and a dedicated WebRTC data channel ([remote teleop](/guides/remote-teleop)). Each frame carries a shared `seq`; the server keeps a monotonic high-water mark and processes each `seq` exactly once, from whichever link delivers it first — so `on_frame` fires once per logical frame, a quiet USB link fails over to Wi-Fi instantly, and a resuming link can't rewind to a stale pose. The high-water mark resets when the last client disconnects and on `disable()`, so a reloaded headset (its counter restarted) isn't gated against a stale value. The WebRTC pose data channel (`almond_axol.vr.control_channel`) shares the same ICE/TURN path as the camera video, so an operator off your tailnet can drive over a relay. ICE servers are read from the environment (`AXOL_TURN_URL` / `AXOL_TURN_USERNAME` / `AXOL_TURN_PASSWORD`; see `almond_axol.vr.ice`) — on a LAN or plain tailnet they're unset and the direct path is used. If the data channel fails to negotiate (or the peer connection drops), the headset re-requests it after a backoff rather than permanently falling back to the WebSocket. ## Server → headset feedback The server can push a state override to all connected headsets at any time using `VRServer.broadcast_text()`. The headset interprets messages of the form `{"type": "state", "value": "saving"}` as a state override that blocks recording controls. `{"type": "state", "value": "error"}` shows an error indicator in the headset UI. `{"type": "state", "value": "data_collection"}` re-enables controls after saving. The `AxolVRTeleop.send_feedback_state(state)` helper wraps this for all `VRState` values. On connect the server also announces its **operating mode** — `{"type": "mode", "value": "teleop" | "data_collection"}` — set via `VRServer.set_mode()`. The web UI uses it to **lock the HUD** to a single mode: [`axol teleop`](/cli/teleop) sends `teleop` (recording controls hidden), while [`axol collect-data`](/cli/collect-data) sends `data_collection` (recording enabled, but no switching back to plain teleop). The message is pushed once per client in the WebSocket accept handler, so a headset that joins mid-session still gets it. During data collection the server likewise pushes the current **episode number** — `{"type": "episode", "value": N}` (1-based) — which the headset renders as an `Episode N` HUD readout. [`axol collect-data`](/cli/collect-data) announces it at the start of each episode via `AxolVRTeleop.send_feedback_episode(episode)`, which both stores the value on the server (`VRServer.set_episode()`, re-sent on connect so a mid-session headset shows the right number) and broadcasts a live update to connected clients. Plain teleop never sets it, so the readout stays hidden. Teleop additionally broadcasts `{"type": "tracking", "value": true|false}` whenever the engage toggle changes, so the headset knows when the robot is being controlled — it uses this to only allow repositioning and resizing the camera screens (a trigger gesture) while tracking is disengaged. ## Camera video to the headset The server can stream camera frames to the headset over **WebRTC**. Register per-camera sources with `set_video_sources()`. A source is either a **pre-encoded** camera (the GPU-resident `gst_zed` pipeline, exposing `subscribe()` / `alive` — its H.264 access units go straight to RTP with no Python encode) or a **raw frame** source. A connected `ZedCamera` (or stereo eye) is registered directly — the relay adapts it — and you can equally pass any object exposing fixed `width` / `height` / `fps` plus `wait_next(after_ts, timeout_ms)`. The manager picks the right WebRTC track per source automatically: ```python theme={null} vr.set_video_sources({ "overhead": overhead_cam, "left_arm": left_cam, "right_arm": right_cam, }) ``` The existing `/ws` channel is reused for SDP signaling (no extra ports): the headset sends `webrtc-request`, the server replies with a `webrtc-offer` carrying one video track per source (labelled by camera name), and the headset answers with `webrtc-answer`. A stereo overhead streams both eyes **packed side-by-side** in one track, `overhead_sbs` — a single decoder session on the headset, since two full-res sessions cap the Quest's render rate — which the VR app renders per-lens through a WebXR media layer; the SDK-fallback path (no gst) instead registers the two per-eye sources `overhead_left` / `overhead_right`. A stereo wrist is registered as a single source under its slot name (one feed, like a mono camera). Dataset **recording** is unaffected — it still stores the two per-eye observation keys. Pass `None` or `{}` to turn video off. [`axol teleop --cameras`](/cli/teleop) and [`collect-data`](/cli/collect-data) wire this up automatically. ### Pipeline Teleop and data collection share one **GPU-resident** camera path (`almond_axol.video.gst_zed`). The zed-gstreamer source elements (`zedxonesrc` mono / `zedsrc` stereo) grab frames straight into NVMM (GPU) memory, and a single in-process GStreamer pipeline tees that one zero-copy buffer to two consumers: * an **encoded** branch — `nvv4l2h264enc` (NVENC) → `appsink` — so the whole grab→encode chain stays on the GPU (\~5–10 ms) and Python only ever sees the H.264 access units, which **aiortc** forwards as pre-encoded packets (`encoder.pack`, no Python encode); * a **raw** branch — `nvvidconv` (the VIC) converts to RGBA and downscales to the dataset resolution — built only when raw frames are needed (the dataset, policy inference). For the dataset it ends in a gst-native **`shmsink`** that exports each frame to the recorder subprocess in C (no Python pull in the relay); for in-process consumers like policy inference it ends in an `appsink` pulled into numpy. Each frame still carries a `capture_perf_ts` aligned to the joint sample (the in-process path maps the buffer PTS onto `time.perf_counter`; the dataset path stamps receipt time minus the relay's measured pipeline latency), so dataset rows align image capture with the joint sample — without the ZED SDK's \~26 ms host round trip. Both **teleop** and **data collection** run the camera grab/encode/WebRTC pump in a **dedicated subprocess** (`almond_axol.video.video_proc.VideoRelayProcess`, registered via `set_video_manager()`): isolating it keeps the control loops at full rate (running it in-process measurably halves the IK rate). For teleop the subprocess opens the cameras with the raw branch off (encoded only). For data collection it opens **both** branches — the encoded branch streams the headset view, and the raw branch is exported to a second dedicated **recorder subprocess** (`almond_axol.recording.record_proc.DatasetRecorderProcess`) that owns the `LeRobotDataset` end to end. On the Jetson that export is a gst-native `shmsink`→`shmsrc` transport, so the relay runs **no Python per raw frame** — keeping its single-threaded aiortc send uncontended during recording — with a `multiprocessing` shared-memory copy (`almond_axol.video.shm_frames`, `RawFrameReader`) as the fallback when gst's `shm` plugin is absent. The control loop only writes a small joint/action snapshot to shared memory each tick (`SnapshotWriter`) and never touches the grab/encode/dataset path; it still receives each camera's dimensions via `AxolRobot.set_external_cameras()` to size the dataset features. Capture timestamps stay aligned because `time.perf_counter` is `CLOCK_MONOTONIC` (a shared origin across processes), so a `capture_perf_ts` stamped in the relay is directly comparable to the joint-sample timestamps. aiortc handles the WebRTC transport (RTP/SRTP/ICE) and SDP signaling in both cases. When the gst stack is unavailable, data collection falls back to the in-process camera path (`set_video_sources()` with the robot owning the cameras), and both paths fall back to the ZED Python SDK grab (`ZedCamera`) plus in-Python NVENC (`almond_axol.video.hw_video`). Requires the GPU-resident camera stack the host installer sets up once: the GStreamer + PyGObject support from [`axol gst.install`](/cli/gst-install) plus the patched zed-gstreamer source plugins from [`axol gst.build-zed`](/cli/gst-build-zed) (the NVENC elements themselves ship with the Jetson L4T BSP). The NVENC/VIC engine clocks must also be pinned to their maximum for low-latency encode ([`axol jetson.setup`](/cli/jetson-setup), run at boot by the installer's systemd unit); the default Jetson devfreq governor scales them for throughput, which otherwise roughly triples per-frame encode latency. ## `VRServerConfig` fields | Field | Default | Description | | -------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `port` | `8000` | WSS listen port | | `certfile` | auto | Path to TLS certificate; `None` uses the auto-generated cert | | `keyfile` | auto | Path to TLS private key | | `interp_enabled` | `True` | Enable the adaptive playout buffer that interpolates pose frames using the headset timestamp `t`, smoothing jittery / batched frame arrivals | | `interp_min_delay_s` | `0.0` | Minimum playout delay in seconds | | `interp_max_delay_s` | `0.1` | Maximum playout delay in seconds — caps how much latency the buffer may add | Before opening the VR app, accept the self-signed certificate by navigating to `https://.local:8000` in the VR browser and proceeding past the security warning.