almond_axol.vr

from almond_axol.vr import VRServer, VRServerConfig

import asyncio
from almond_axol.vr import VRServer, VRServerConfig

async def main():
    async with VRServer(VRServerConfig(port=8000)) as vr:
        while True:
            frame = vr.get_frame()
            if frame is not None:
                print(frame.l_ee, frame.r_ee)
            await asyncio.sleep(0.01)

asyncio.run(main())

def on_frame(frame):
    print(frame.l_grip, frame.r_grip)

async with VRServer() as vr:
    vr.set_on_frame(on_frame)
    await asyncio.sleep(float("inf"))

​

VRFrame fields

​

Controller input

​

Timing and pose source

​

Mantis episode control

​

Set by the server

​

Pose transports and USB/Wi-Fi fallback

• the network WebSocket,
• a USB adb reverse tunnel (Quest over USB),
• a WebRTC data channel (remote teleop).

• Managed Quest Mantis sets VRServerConfig.pose_source_kind="webxr".
• Managed Lighthouse or Ultimate Mantis sets it to "tracker", and gives the bridge and server a one-run expected_pose_source_id.
• Without a pose source policy, the first producer takes control. A tracker bridge can take over from a Quest. Once a tracker has control, the server doesn’t fall back to a connected Quest if the bridge stops.

​

Server to headset feedback

{
  "type": "joints",
  "value": {
    "q": {"<urdf joint>": 0.0},
    "l_grip": 0.0,
    "r_grip": 0.0,
    "engaged": false,
    "pair": {"aligned": false, "width": 0.3, "tilt": 0.0, "grasp": "straight"}
  }
}

• q holds measured joint positions in radians when the robot reports them, and commanded targets otherwise. l_grip and r_grip run from 0 to 1.
• pair is the gripper pair geometry from IKWorker.pair_status(), or null.
  • aligned is true when the grippers already form a box-mode pair. The Box button then reads (aligned), since switching to box mode needs no blend.
  • width is the grip width in metres, between the contact faces.
  • tilt is the pair’s inward tilt in degrees, and grasp is "flush" or "straight".

​

Live session settings

• Client to server: {"type": "set", "key": "<key>", "value": <value>}. The callback set with VRServer.set_on_setting() handles it. The value is checked for type, range and choices, and a rejected request is logged and not echoed. A boolean key also accepts "value": "toggle", which flips the server’s current value.
• Client to server: {"type": "session-config-request"}. The server re-sends that client the session config it sends on connect: mode, settings, pose_source_kind, pose_mode, tracking, episode and hud. Send it once your message listeners are ready, so you don’t miss the copy sent on connect.
• Server to clients: {"type": "settings", "value": {"schema": [...], "values": {"<key>": ...}}}. Sent on connect, on session-config-request, and after every set, so all clients show the server’s state. Each schema entry has key, label, type ("boolean", "select" or "number"), help, options, min, max, step and unit. Both UIs build their controls from it.

​

Camera video to the headset

vr.set_video_sources({
    "overhead":  overhead_cam,
    "left_arm":  left_cam,
    "right_arm": right_cam,
})

• a pre-encoded camera from the GPU pipeline (gst_zed), with subscribe() and alive. Its H.264 frames go straight to the headset with no encoding in Python.
• a connected ZedCamera or stereo eye, which the relay adapts.
• any object with fixed width, height and fps, and wait_next(after_ts, timeout_ms).

• A stereo overhead camera streams both eyes side by side in one track, overhead_sbs, which the VR app shows per eye. Without GStreamer, the SDK fallback sends two tracks, overhead_left and overhead_right.
• A stereo wrist camera streams one feed under its slot name, like a mono camera.
• Dataset recording still stores both eyes of a stereo overhead camera as two observation keys.

​

Pipeline

• Headset: 30 fps, encoded by NVENC (nvv4l2h264enc). aiortc forwards the H.264 packets to the headset without re-encoding.
• Dataset: paced to the recording fps, downscaled and NVENC-encoded. Each frame keeps its exposure timestamp, so the recorder can pair it with the nearest robot state without copying or re-encoding the frame.
• Raw: converted to RGBA for policies and for the fallback path. Its exposure timestamps are also on the time.perf_counter clock.

• a timestamp is missing,
• a frame is stale or repeated,
• the cameras and robot state are too far apart in time,
• the predictive video stream breaks.

• GStreamer and PyGObject from axol gst.install.
• The patched zed-gstreamer plugins from axol gst.build-zed. The NVENC elements come with the Jetson L4T BSP.
• NVENC and VIC clocks pinned to their maximum. axol provision does this (see jetson.setup), and the installer’s systemd unit reapplies it at boot. Without it, encode latency per frame roughly triples.

​

VRServerConfig fields