VR-Mess/AGENTS.md

# VR Messs

VR anime-character "scene viewer" (Koikatsu-VR-class, but with real physics hands).
Not a game: no gameplay loops, quests, or progression. Gameplay features are the
owner's domain; explore, interact, touch, pose — that's the product.

TERMINOLOGY (owner-corrected, use precisely): **"getting a body"** = the physical rig
(torso/arms/hands — the interaction body). **"embodying"** = literally viewing from the
perspective of a character in an animation (the view-follow + avatar system). They are
separate systems; never call the rig milestone "embodying".

This repo is the experimentation lab (hence the name). The designs below are working hypotheses
to be validated with cheap, throwaway spikes — a failed experiment is progress, not rework. The
embodiment/view system in particular is theory until proven.

## Toolchain pins (do not drift)

- **Godot 4.7.2** (current stable, Forward Plus renderer), **C# / .NET 8**. Desktop-only (PCVR).
- **Physics: Box3D** via the `godot-box3d` GDExtension (server-replacement package by
  bearlikelion) — **installed, v0.2.6**, setting value `physics/3d/physics_engine="Box3D
  Physics"`. Validation spike (`scenes/physics_spike.tscn`, run headless) PASSES: drop+contact
  and velocity-chase-into-wall both work from C#. WARNING: exactly ONE `.gdextension` file may
  exist — a stray duplicate (e.g. at `addons/` root) double-loads the library and breaks the
  server with "required virtual method" errors. **We own a fork** —
  `https://github.com/Foxyz1302/godot-box3d` (branch `rigid-weld-joint`, carries the weld
  patch; PR to upstream pending, owner-side) — and it may gain more additions in the future
  (rebuild recipe + source in `.vendor/`, only the Linux binary is patched locally).
  Jolt (built-in) is the fallback: reverting is
  the one project-setting line, and all gameplay code must stay standard-Godot-physics-node
  based so that flip is lossless.
- Character import (Koikatsu etc.) is a separate pipeline owned by someone else. We consume a
  character scene that provides: `Skeleton3D` with humanoid bone mapping + `AnimationPlayer`.
  Don't touch import code; if the interface is insufficient, say so instead of hacking around it.
- Dev runtime stack: **WiVRN + WayVR** on Linux. Headset and runtime both support hand tracking —
  treat hand tracking as available. View-follow milestone verified working (OpenXR + WiVRN).
- Graphics are deliberately **stylized** — fits the anime look and buys performance headroom.
  Renderer choice (currently Forward+; docs recommend Mobile for desktop VR) is deferred until
  profiling justifies a change.

## Documentation rules (hard rules — this project's #1 failure mode)

- Local pinned docs: `/home/foxyz/Projects/Games/.shared/godot-docs-4.7` — full manual +
  class reference in `.rst` (source of docs.godotengine.org, branch `4.7`). Grep it first.
- For ANY engine API detail, class name, property, or signal: **trust only these 4.7 docs or a
  live spike.** Training-memory APIs and older-version knowledge are banned — verify or don't write it.
- Web fallback: `https://docs.godotengine.org/en/4.7/...` only — the version goes IN the URL.
  Never `/latest/`, never another version path.
- Code examples: rst sources contain `code-tab::` blocks with BOTH gdscript and csharp — grep for
  `code-tab:: csharp`. Class reference lives in `classes/class_*.rst`.
- XR tutorials: `tutorials/xr/` (setting_up_xr, xr_action_map, xr_room_scale,
  openxr_body_tracking, basic_xr_locomotion). Physics: `tutorials/physics/` (rigid_body,
  ragdoll_system, using_jolt_physics, physics_introduction).

## Godot/C# traps

- Addons that are GDScript-only can't be referenced from C# — **Godot XR Tools is GDScript-only**:
  treat it as a reference implementation to port ideas from, never a dependency.
- Default OpenXR action map names (verified from engine source, 4.7): `trigger`/`grip` (Float),
  `ax_button` (A/X), `by_button` (B/Y), `menu_button` (menu/system — often eaten by the runtime,
  e.g. WiVRN screenshot+anchor-hold), `primary` (thumbstick). `select_button` is Pico-only.
  Poses: `default_pose`, `aim_pose`, `grip_pose`, `palm_pose`.
- C# in Godot covers desktop + mobile only (fine — PCVR target).
- Signals: C# `+=`/`-=` events; check the class ref for exact C# signatures.
- The C# binding can diverge from engine class names: `AudioStreamWAV` is `AudioStreamWav`
  in C#. When docs and compiler disagree, grep the shipped `GodotSharp.dll` (strings) —
  it's at `/usr/lib/godot-mono/GodotSharp/Api/Debug/GodotSharp.dll` on this machine.

## Architecture (working hypotheses — test cheap before committing)

Four layers, bottom-up, applied per frame in this order:

1. **Animation** — `AnimationPlayer` drives the skeleton (any pose, any animation).
2. **Tracked override / view embodiment** — several input modes, all blended over whatever
   animation is playing:
   - **View-follow (core feature, Koikatsu-VR-style)**: the view follows the character's head
     frame including tilt, so weird animations stay correct. The player's HMD look is remapped
     through a one-time calibration transform captured relative to the character's chest
     ("chest calibration", e.g. once while lying flat) and drives a look override on the
     head+neck (possibly slight chest) bones. Skeleton-space only — the physics layer stays
     separate from this. Pivot behavior (rotation-only snap vs positional head offset) and the
     calibration UX are open experiments.
   - **Free-view**: ordinary playspace (`XROrigin3D`); "lean back mode" = origin offset +
     recline calibration, zero physics involved.
   - **Single-controller mode**: one controller as master tracker — hand override, or strapped
     to the hip driving the animation root/hip (hip-tracker mode).
   - Full-body trackers are a far-future extension of this same layer (more override targets),
     not a separate system.
3. **Physics embodiment** — a small set of rigid bodies (hand L/R, head, torso/hip capsule) that
   CHASE layer-1/2 targets: velocity-clamped steering (`v = clamp((target - pos) / dt)`), CCD on,
   no joints between them. Unobstructed → effectively 1:1 tracking; obstructed → pushes with
   tunable strength. The chase deviation is the feedback signal: it clamps the visual hand offset,
   pushes the character pose via spring-damped offsets ("pushable from animation"), and fires
   logical touch-reaction events. Boneworks calls the general technique "heuristic force limit".
4. **Secondary motion (hair/skirt/cloth) — custom spring-bone solver.** Koikatsu DynamicBones and
   VRM springBone are poorly interactive and/or expensive; VRChat PhysBones are the gold standard
   but proprietary. So: original PhysBones-*style* math (spring-damper chains on the skeleton,
   animation-space) with analytic sphere/capsule colliders (body, hands, floor). The physics
   engine contributes only its QUERY API (throttled ray/shape casts) so bones respect scene
   geometry — **never rigid-body chains for jiggle**. Performance is a first-class goal of this
   project: chain budgets, screen-size/distance LOD, simplified colliders. If the per-bone C#
   loop profiles hot, the hot loop moves to a GDExtension — profile first, don't pre-optimize.

Do NOT build anything that needs 6DOF/cone-twist joints between tracked bodies (the wrapper lacks
them; velocity steering replaces them). Scene objects stay standard nodes (`RigidBody3D`,
`StaticBody3D`, `CharacterBody3D`) so arbitrary imported scene content works and the Jolt fallback
stays real.

## XR

OpenXR is built into Godot 4.7 — no plugin. Node set: `XROrigin3D` / `XRCamera3D` /
`XRController3D` (+ `XRBodyTracker`/`XRHandTracker` for body/hand tracking data). OpenXR runs on
device only; the editor has an XR simulator for dev. Trackers/action map: see `xr_action_map.rst`.

## Interaction & UI (design directions — experiments welcome)

- **Touch-first hybrid cursor**: interactive surfaces prefer *touch*. A pointer tip (controller
  tip or index fingertip) acts as a cursor when in proximity of a surface — push, hold, slide,
  physical press feel. Out of proximity, fall back to a laser ray for distant precision. Lasers
  ruin immersion and are hard to aim at distance; they are the fallback, not the default.
- **Hand tracking**: cheap spike early, full support later. Interaction logic must not assume
  controllers — controller tip and fingertip feed the same cursor concept.
- **Locomotion** (implemented, NEEDS FIXING per owner 2026-10-01 — two-hand turn not
  2-anchor based, two-hand simultaneous move weird, teleport + lasers broken/experimental;
  fix-or-replace rides on the body-physics track decision, owner candidate replacement =
  physics thumbstick movement):
  spawn re-anchoring
  happens ONCE on first focus gained (HMD xz at `SpawnPoint` + spawn yaw; real head height and
  pitch kept) — physical room position never matters. `pose_recentered` is deliberately NOT a
  placement trigger AND its log is throttled to one line: measured on WiVRN it fires
  continuously at refresh rate (~90 Hz, 22k lines/session) — reacting to it was the original
  view-locked-forward bug. `Locomotion.cs` is the manager (spawn placement,
  teleport markers, lasers, fade); **grab movement lives in `AnchorMoveMode.cs` as its own
  movement mode** — scene-viewer-first, other modes (walk, tp-only variants) are future nodes
  beside it. Teleport markers (`scenes/teleport_marker.tscn`):
  aim + trigger, with fade; near markers (within `MarkerNearDistance` 1.2 m) fully hide
  (disc, beam, label) AND are not teleport-targetable. Grab semantics are
  ANCHOR-based: hands are anchors to push/pull from — one-hand grip
  moves the player OPPOSITE to hand motion in full 3D, both grips = anchor translation +
  IN-PLACE yaw turn around the head. All grab deltas and turn pivots are computed in
  WORLD space (tracking-space math made turns orbit the map center — do not regress).
  Spawn/teleport placement pins arrival height (head height preserved). Locomotion drives
  `XROrigin3D` directly; physics reconciliation comes with the embodiment layer. Grip/trigger
  are polled Floats with thresholds per docs
  (bool actions unreliable across runtimes). Engine logs land in
  `~/.local/share/godot/app_userdata/VR Messs/logs/godot.log` — agents: read this after user
  playtests instead of guessing.
- **UI direction — physical prop controls first**: buttons, knobs, sliders as real 3D objects
  (touch interaction, later Box3D for feel). Godot has NO native world-space widget system —
  `Control`s are screen-space; world UI is always Control-rendered-into-3D via `SubViewport`
  (optionally `SubViewportCompositionLayer` for XR-native sharpness/perf). Use at most one
  SubViewport/composition-layer screen for dense content (file browser, settings) if needed;
  everything else is props. World-space text is native and cheap: `Label3D` (SDF, crisp at
  distance) and `TextMesh` (extruded) — no SubViewport required.
- **`WorldPanel.cs` is the first prop-UI prototype**: summoned/dismissed via `ax_button` (A/X —
  menu_button belongs to the runtime). It is PARENTED UNDER `XROrigin3D` — Koikatsu-VR-style
  "fixed in the player's room": IRL movement doesn't move it relative to you, game locomotion
  carries it. If that feel is wrong, reverting = reparenting to `Main` and fixing the `_Ready`
  node paths. Floats in front of the summoning hand facing the player (full 3D facing — works
  from lying poses), draggable directly or via laser at fixed ray distance while gripped
  (continuously re-orients toward the head). The Spawn
  knob also recalls the panel in front of the head. The pointer
  tip is a protruding stylus cone (apex ~12 cm ahead of the controller) and is THE cursor:
  knobs hover-highlight and PRESS ON CONTACT (tip within 4.5 cm, with cooldown) — physical
  buttons, no trigger needed; trigger still presses laser-hovered knobs (knobs are
  laser-targetable). Presses play a synthesized click (code-generated `AudioStreamWav` PCM
  — no asset). Spawn/Ping/Close knobs. Locomotion defers
  grab/teleport to the panel when a hand is near it (`WorldPanel.IsBlocking`). Laser lines are
  owned by `Locomotion.UpdatePointer`: shown only when aiming at something interactive (marker
  or panel) AND beyond `LaserMinDistance` (0.35 m) — near targets are touched via the pointer
  tip instead, which is the touch-first hybrid cursor in miniature. Grab deltas and turn
  pivots are computed in WORLD space (tracking-space math made two-hand turns orbit the map
  center — do not regress this).
- **Control scheme is deliberate**: decide what lives on controllers vs. what lives in menus
  before stacking bindings — controller button budget is scarce, menus carry the rest.

## Order of work

1. ~~Get VR working at all~~ — **done**: OpenXR + WiVRN verified in-headset, flat preview
   fallback works without a headset.
2. ~~Input foundation (partial)~~ — grip/trigger Float polling + `ax_button` summon verified
   from engine source; the deliberate full action-map pass happens with the UI milestone.
   **Hand tracking spike still TODO** (`XRHandTracker` through WiVRN, see
   `openxr_hand_tracking.rst`).
3. ~~Locomotion: anchor-grab mode~~ — **done, NEEDS FIXING** (`AnchorMoveMode.cs` + teleport
   markers + spawn anchoring): two-hand turning not 2-anchor based, simultaneous two-hand
   movement weird, teleport + lasers broken (owner report 2026-10-01; experiments, not
   current issues). Fix-or-replace deferred — owner's candidate replacement is physics
   thumbstick movement if the body-physics track lands.
4. ~~UI: first prop panel~~ — **done** (`WorldPanel.cs`); SubViewport/composition-layer screen
   deferred until dense content (file browser) is actually needed.
5. ~~Box3D validation spike~~ — **done, PASS** (drop+contact, velocity-chase vs wall, headless
   via `scenes/physics_spike.tscn`).
6. **Physics hands & props sandbox — grab v5.3 = SOLVER-OWNED RIGID WELD (patched addon).**
   Research first (see `research/vr-grabbing.md`, `vr-two-handed-grab.md` + its dated
   CORRECTION, `godot-box3d-weld.md` — sourced taxonomy, Box3D internals, the whole v5.0→5.2
   history). The owner rejected every point-joint variant as "hanging on a winch" (ball
   orbits the palm, anvil surface drifts, palms clip) — a grab must lock the object's POSE
   relative to the hand, not one point + a rotator. **The addon is now locally patched**
   (source + recipe in `.vendor/`, originals in `addons/godot-box3d/bin-stock/`, Linux binary
   only): Godot's ConeTwistJoint3D maps to Box3D's `b3WeldJoint`, pre-corrected so the weld
   locks the relative pose AT CREATION (no identity-target snap; bias semantics INVERTED vs
   pin — 0 = max stiffness). `PropGrabber.cs`: grab creates a ConeTwistJoint3D weld at the
   palm point (`SwingSpan/TwistSpan 0`, `ExcludeNodesFromCollision`, configured BEFORE
   AddChild — mandatory ordering). NO script forces/torques/damp anywhere and NO per-tick
   joint re-positioning (the weld is not a hand-following pivot) — the solver owns everything
   and weight flows into the hand's 90 N budget honestly. BOTH hands weld RIGID (bias 0 —
   v5.3.1: the owner rejected the soft secondary; hand₁+prop+hand₂ welded = one composite
   body, conflicting controller demands resolve through hand lag, not object stretch);
   takeover is automatic — the survivor's weld persists, and the brief ~3° ring when one
   hand releases is stored weld stress discharging (honest physics). Break: stretch-based (rip 0.8 m / 0.25 m for 0.25 s / both-hands
   0.4 m) + "grip dragged off" when the hand body lags its controller > `PhysicsHands.SnapDistance`.
   Headless regressions: `scenes/weld_spike.tscn` (addon patch), `scenes/physics_spike.tscn`,
   `scenes/grab_spike.tscn` (rigid pose under drag + fast spin — the winch regression; don't
   tighten rotation limits below ~2°, that's the gravity floor for off-center grabs).
   **Grab v5.3.1 VALIDATED in-headset (owner: "it just works") — grab saga closed.**
   `PhysicsHands.cs`: physical palm bodies (0.4 kg, 90 N chase, rotation lock, stuck-safety
   snap — SUSPENDED for a hand while it grips, HAND ghosts > 10 cm lag). Collision layers:
   **1 = world+props, 2 = UI, 3 = player body (PhysicsTorso chest), 4 = player hands,
   5 = player arms (PhysicsArms segments)** — no self-collision between own body parts
   (Nimso/VRChat precedent); props/floor/table mask 1|3|4|5, hands 4/1|4, chest 3/1|3,
   arm segments 5/1; panel stays 2/0 (props pass
   through even when shown); layers live on
   CollisionObject3D, NOT CollisionShape3D. Panel = prop-UI prototype (ax_button summon,
   contact-press knobs, Reset). Laser/distant grab REMOVED (user never knew it existed);
   grab = palm-proximity only, per-hand palm-face offset (left +X, right −X — taste).
   PinJoint quirks (anchors captured at creation; joint node position READ every tick;
   break measures the prop-local anchor vs the palm point) apply to the v5.0–5.2 PIN path
   only — the weld does not share the pivot behavior. Research reports live in `research/`
   (arm IK, IK stability/deep, grabbing, two-handed grab/Box3D internals, addon weld patch,
   body physics hugs, body physics Godot facts) — READ before re-researching or changing
   grab/IK/body. Stink-O's box3d-godot (custom components) = documented plan-B.
8. **Body physics Stage 1 — PHYSICAL CHEST, implemented 2026-10-01, FAILED first playtest.**
   `PhysicsTorso.cs`:
   one chase-driven RigidBody3D (30 kg box, gravity off, CCD) toward an HMD-derived target —
   the PhysicsHands PD pattern at body mass (2400 N budget), orientation = slewed
   angular-velocity servo (self-rights, holds yaw), 2 m snap safety, torso-yaw spring moved
   here from BodyRig. Camera stays KINEMATIC — torso chases the head, never the reverse
   (Boneworks sickness lesson). No self-collision with own hands (layer scheme above).
   **First in-headset session NaN'd the chest — ROOT-CAUSED + FIXED same day**: script
   math, not the solver (`Quaternion.GetAngle()` = 2·acos(W) → NaN at W = 1.0000001; see
   `research/body-stage1-failure-brief.md` RESOLUTION). **Never use `GetAngle()/GetAxis()`
   for servo errors — `BodyMath.RotationError` (atan2) is the rule; every driven body runs
   `BodyMath.ResetIfNonFinite`.** Regression: `scenes/collapse_spike.tscn` (run headless with
   `--fixed-fps 90`) must print "COLLAPSE: overall PASS". `BodyRig.cs` is
   VISUALS-only (head cube + arm boxes drawn through the physical elbow; IK + reach budget
   moved to `PhysicsArms`, item 10). Headless regression:
   `scenes/body_spike.tscn` must print "BODY: overall PASS" (also proves the Box3D pair
   rules: one-way contact, hand-through-chest exclusion, prop contact) — note its repro gap:
   no static-table overlap, no welded heavy props. Stage 2 (physical
   arm chains + hierarchical per-joint budgets, Nimso/Boneworks tier) — see item 10; design + prior art in `research/vr-body-physics-hugs.md`,
   stack facts in `research/vr-body-physics-godot.md`.
9. **Getting a body (rig) — placeholder v1.3 (cube character, agent-built).** (The IK/yaw/reach
   math described here now lives in `PhysicsArms.cs` / `PhysicsTorso.cs`; `BodyRig.cs` only
   draws.) Original notes:
   all-BOX primitives (owner preference) — head 0.15³ behind the eyes, slim torso retracted,
   wrist-anchored 2-bone arms. **Soft IK applied**: shoulder→hand distance clamped to
   reach×0.98 before the solve (the straight-arm singularity was the overextension shake; if it
   persists, lower SoftIKFactor toward 0.95). Elbow pole: torso frame, shoulder→hand axis
   PROJECTED OUT, hand weight 0.25, exp-smoothed 10/s (twitchy wrists fixed). **Torso yaw
   spring**: HMD yaw base blended toward hands-derived yaw (weight 0.6 × forwardness over
   0.3 m), exp-smoothed 12/s, decoupled from the head — the "chest = headset + hands" ask.
   Reach budget hysteresis unchanged. Full sourced findings in `research/vr-arm-ik.md` and
   `research/vr-arm-ik-deep.md` (HL:Alyx went hands-only on purpose; SkeletonIK3D deprecated →
   custom SkeletonModifier3D when a real skeleton arrives; LookAtModifier3D for head→neck).
   EMBODYING (view-from-character-perspective) comes after
   the head/neck look override exists. **HANDOFF.md at project root is the takeover document**
   (constant table verified against code — code wins on drift); keep it updated when systems
   change.
10. **Body physics Stage 2 — PHYSICAL ARMS, implemented 2026-10-02, awaiting owner playtest.**
   `PhysicsArms.cs`: per side an upper-arm + forearm RigidBody3D (2.0 / 1.2 kg, layer 5) that
   PD-chase their analytic 2-bone IK pose, solved each physics tick from the PHYSICAL chest
   shoulders to the PHYSICAL hands. **No joints** — Box3D has no usable 3-DOF shoulder/wrist
   joint (PinJoint's hidden identity spring, no Generic6DOF) and this file's architecture
   forbids joints between tracked bodies; the research's hinge-motor chain is therefore
   deferred, chase segments are the Box3D-honest version. Hierarchical budgets (Nimso muscle
   model) from a 60 kg body-mass stat: hand 90 N ≺ forearm 180 N ≺ upper arm 360 N ≺ chest
   2400 N. What it gives: arms that push props, rest on / get blocked by geometry (no
   tunneling), and draw honestly (BodyRig's elbow = the physical elbow). What it does NOT do
   yet (Stage 2b candidates, owner feel decides): a blocked forearm does not hold the HAND
   back (hands still chase the controllers independently), and prop/NPC forces on the arm
   don't flow into the chest. Regression: `scenes/arm_spike.tscn` "ARM: overall PASS";
   the full body also runs inside `scenes/collapse_spike.tscn`. Design + prior art:
   `research/vr-body-physics-hugs.md` §5–7, `research/vr-body-physics-godot.md`.