VR-Mess/AGENTS.md

VR Messs

VR anime-character "scene viewer" (Koikatsu-VR-class, but with real physics hands). Not a game: no gameplay loops, quests, or progression. Gameplay features are the owner's domain; explore, interact, touch, pose — that's the product.

TERMINOLOGY (owner-corrected, use precisely): "getting a body" = the physical rig (torso/arms/hands — the interaction body). "embodying" = literally viewing from the perspective of a character in an animation (the view-follow + avatar system). They are separate systems; never call the rig milestone "embodying".

This repo is the experimentation lab (hence the name). The designs below are working hypotheses to be validated with cheap, throwaway spikes — a failed experiment is progress, not rework. The embodiment/view system in particular is theory until proven.

Toolchain pins (do not drift)

• Godot 4.7.2 (current stable, Forward Plus renderer), C# / .NET 8. Desktop-only (PCVR).
• Physics: Box3D via the godot-box3d GDExtension (server-replacement package by bearlikelion) — installed, v0.2.6, setting value physics/3d/physics_engine="Box3D Physics". Validation spike (scenes/physics_spike.tscn, run headless) PASSES: drop+contact and velocity-chase-into-wall both work from C#. WARNING: exactly ONE .gdextension file may exist — a stray duplicate (e.g. at addons/ root) double-loads the library and breaks the server with "required virtual method" errors. We own a fork — https://github.com/Foxyz1302/godot-box3d (branch rigid-weld-joint, carries the weld patch; PR to upstream pending, owner-side) — and it may gain more additions in the future (rebuild recipe + source in .vendor/, only the Linux binary is patched locally). Patches so far: ConeTwistJoint3D → rigid weld (research/godot-box3d-weld.md), Generic6DOFJoint3D → ball/weld/drive with torque-capped muscle motors (research/godot-box3d-6dof.md, regression scenes/sixdof_spike.tscn). Jolt (built-in) is the fallback: reverting is the one project-setting line, and all gameplay code must stay standard-Godot-physics-node based so that flip is lossless.
• Character import (Koikatsu etc.) is a separate pipeline owned by someone else. We consume a character scene that provides: Skeleton3D with humanoid bone mapping + AnimationPlayer. Don't touch import code; if the interface is insufficient, say so instead of hacking around it.
• Dev runtime stack: WiVRN + WayVR on Linux. Headset and runtime both support hand tracking — treat hand tracking as available. View-follow milestone verified working (OpenXR + WiVRN).
• Graphics are deliberately stylized — fits the anime look and buys performance headroom. Renderer choice (currently Forward+; docs recommend Mobile for desktop VR) is deferred until profiling justifies a change.

Documentation rules (hard rules — this project's #1 failure mode)

• Local pinned docs: /home/foxyz/Projects/Games/.shared/godot-docs-4.7 — full manual + class reference in .rst (source of docs.godotengine.org, branch 4.7). Grep it first.
• For ANY engine API detail, class name, property, or signal: trust only these 4.7 docs or a live spike. Training-memory APIs and older-version knowledge are banned — verify or don't write it.
• Web fallback: https://docs.godotengine.org/en/4.7/... only — the version goes IN the URL. Never /latest/, never another version path.
• Code examples: rst sources contain code-tab:: blocks with BOTH gdscript and csharp — grep for code-tab:: csharp. Class reference lives in classes/class_*.rst.
• XR tutorials: tutorials/xr/ (setting_up_xr, xr_action_map, xr_room_scale, openxr_body_tracking, basic_xr_locomotion). Physics: tutorials/physics/ (rigid_body, ragdoll_system, using_jolt_physics, physics_introduction).

Godot/C# traps

• Addons that are GDScript-only can't be referenced from C# — Godot XR Tools is GDScript-only: treat it as a reference implementation to port ideas from, never a dependency.
• Default OpenXR action map names (verified from engine source, 4.7): trigger/grip (Float), ax_button (A/X), by_button (B/Y), menu_button (menu/system — often eaten by the runtime, e.g. WiVRN screenshot+anchor-hold), primary (thumbstick). select_button is Pico-only. Poses: default_pose, aim_pose, grip_pose, palm_pose.
• C# in Godot covers desktop + mobile only (fine — PCVR target).
• Signals: C# +=/-= events; check the class ref for exact C# signatures.
• The C# binding can diverge from engine class names: AudioStreamWAV is AudioStreamWav in C#. When docs and compiler disagree, grep the shipped GodotSharp.dll (strings) — it's at /usr/lib/godot-mono/GodotSharp/Api/Debug/GodotSharp.dll on this machine.

Architecture (working hypotheses — test cheap before committing)

Four layers, bottom-up, applied per frame in this order:

1. Animation — AnimationPlayer drives the skeleton (any pose, any animation).
2. Tracked override / view embodiment — several input modes, all blended over whatever animation is playing:
   • View-follow (core feature, Koikatsu-VR-style): the view follows the character's head frame including tilt, so weird animations stay correct. The player's HMD look is remapped through a one-time calibration transform captured relative to the character's chest ("chest calibration", e.g. once while lying flat) and drives a look override on the head+neck (possibly slight chest) bones. Skeleton-space only — the physics layer stays separate from this. Pivot behavior (rotation-only snap vs positional head offset) and the calibration UX are open experiments.
   • Free-view: ordinary playspace (XROrigin3D); "lean back mode" = origin offset + recline calibration, zero physics involved.
   • Single-controller mode: one controller as master tracker — hand override, or strapped to the hip driving the animation root/hip (hip-tracker mode).
   • Full-body trackers are a far-future extension of this same layer (more override targets), not a separate system.
3. Physics embodiment — a small set of rigid bodies (hand L/R, head, torso/hip capsule) that CHASE layer-1/2 targets: velocity-clamped steering (v = clamp((target - pos) / dt)), CCD on, no joints between them. Unobstructed → effectively 1:1 tracking; obstructed → pushes with tunable strength. The chase deviation is the feedback signal: it clamps the visual hand offset, pushes the character pose via spring-damped offsets ("pushable from animation"), and fires logical touch-reaction events. Boneworks calls the general technique "heuristic force limit".
4. Secondary motion (hair/skirt/cloth) — custom spring-bone solver. Koikatsu DynamicBones and VRM springBone are poorly interactive and/or expensive; VRChat PhysBones are the gold standard but proprietary. So: original PhysBones-style math (spring-damper chains on the skeleton, animation-space) with analytic sphere/capsule colliders (body, hands, floor). The physics engine contributes only its QUERY API (throttled ray/shape casts) so bones respect scene geometry — never rigid-body chains for jiggle. Performance is a first-class goal of this project: chain budgets, screen-size/distance LOD, simplified colliders. If the per-bone C# loop profiles hot, the hot loop moves to a GDExtension — profile first, don't pre-optimize.

SUPERSEDED for the player (2026-10-02): the fork now maps Generic6DOFJoint3D (ball/drive/weld), and the player body is a jointed muscle ragdoll (item 11). Layer 3 above describes the legacy chase bodies, which remain valid for non-player things (and their spikes). Scene objects stay standard nodes (RigidBody3D, StaticBody3D, CharacterBody3D) so arbitrary imported scene content works and the Jolt fallback stays real.

XR

OpenXR is built into Godot 4.7 — no plugin. Node set: XROrigin3D / XRCamera3D / XRController3D (+ XRBodyTracker/XRHandTracker for body/hand tracking data). OpenXR runs on device only; the editor has an XR simulator for dev. Trackers/action map: see xr_action_map.rst.

Interaction & UI (design directions — experiments welcome)

• Touch-first hybrid cursor: interactive surfaces prefer touch. A pointer tip (controller tip or index fingertip) acts as a cursor when in proximity of a surface — push, hold, slide, physical press feel. Out of proximity, fall back to a laser ray for distant precision. Lasers ruin immersion and are hard to aim at distance; they are the fallback, not the default.
• Hand tracking: cheap spike early, full support later. Interaction logic must not assume controllers — controller tip and fingertip feed the same cursor concept.
• Locomotion (implemented, NEEDS FIXING per owner 2026-10-01 — two-hand turn not 2-anchor based, two-hand simultaneous move weird, teleport + lasers broken/experimental; fix-or-replace rides on the body-physics track decision, owner candidate replacement = physics thumbstick movement): spawn re-anchoring happens ONCE on first focus gained (HMD xz at SpawnPoint + spawn yaw; real head height and pitch kept) — physical room position never matters. pose_recentered is deliberately NOT a placement trigger AND its log is throttled to one line: measured on WiVRN it fires continuously at refresh rate (~90 Hz, 22k lines/session) — reacting to it was the original view-locked-forward bug. Locomotion.cs is the manager (spawn placement, teleport markers, lasers, fade); grab movement lives in AnchorMoveMode.cs as its own movement mode — scene-viewer-first, other modes (walk, tp-only variants) are future nodes beside it. Teleport markers (scenes/teleport_marker.tscn): aim + trigger, with fade; near markers (within MarkerNearDistance 1.2 m) fully hide (disc, beam, label) AND are not teleport-targetable. Grab semantics are ANCHOR-based: hands are anchors to push/pull from — one-hand grip moves the player OPPOSITE to hand motion in full 3D, both grips = anchor translation + IN-PLACE yaw turn around the head. All grab deltas and turn pivots are computed in WORLD space (tracking-space math made turns orbit the map center — do not regress). Spawn/teleport placement pins arrival height (head height preserved). Locomotion drives XROrigin3D directly; physics reconciliation comes with the embodiment layer. Grip/trigger are polled Floats with thresholds per docs (bool actions unreliable across runtimes). Engine logs land in ~/.local/share/godot/app_userdata/VR Messs/logs/godot.log — agents: read this after user playtests instead of guessing.
• UI direction — physical prop controls first, ONE dense screen: buttons, knobs, sliders as real 3D objects where they live in the world (touch interaction, later Box3D for feel). Godot has NO native world-space widget system — Controls are screen-space; world UI is always Control-rendered-into-3D via SubViewport (optionally SubViewportCompositionLayer for XR-native sharpness/perf). At most ONE SubViewport/composition-layer screen for dense content — that screen is now the in-VR menu (WorldPanel, below); everything else is props. World-space text is native and cheap: Label3D (SDF, crisp at distance) and TextMesh (extruded) — no SubViewport required.
• WorldPanel.cs = the in-VR menu (2026-10-02, replaced the chunky prop-knob panel): real Godot Control UI (MainMenu.cs: tabs Player / World / System, normal-sized buttons, slider, scrollable log, FPS) rendered by MenuScreen.cs into a 960×640 SubViewport shown on a thin 0.36×0.24 m panel (~2.7 px/mm ≈ 1:1 with headset pixels at 45 cm — viewport textures have no mipmaps, so don't oversample). Summoned/dismissed via ax_button (A/X — menu_button belongs to the runtime). It is PARENTED UNDER XROrigin3D — Koikatsu-VR-style "fixed in the player's room": IRL movement doesn't move it relative to you, game locomotion carries it. If that feel is wrong, reverting = reparenting to Main and fixing the _Ready node paths. Appears 0.3 m in front of the summoning hand facing the head (full 3D facing — works from lying poses), draggable by grip directly or via laser at fixed ray distance (continuously re-orients toward the head); "Return to spawn" recalls it in front of the head. Input = the touch-first hybrid cursor, injected as mouse events: each frame both controller rays are projected onto the panel plane (analytic, no physics query), converted to SubViewport pixels and pushed with Viewport.PushInput(event, true) (InputEventMouseMotion/InputEventMouseButton, NotifyMouseEntered/Exited). The stylus tip (11 cm ahead of the controller = the drawn stylus tip, which sticks out of the FRONT of the hand box from 6 cm; owner: it used to read as mid-palm) within 6 cm of the surface = TOUCH cursor: crossing 0.6 cm presses, backing out past 1.8 cm releases (hysteresis; entering from the side/behind never presses) — buttons fire on release like a real tap, sliders drag while touching. Farther away = LASER cursor, trigger clicks/drags (edge-triggered: a trigger already held never clicks). Thumbstick Y while hovering = mouse wheel (scroll containers); touch on a scrollable area = swipe-vs-tap (press held back, > 6 mm travel scrolls + flings, else a tap is sent at the contact point). One Godot mouse per viewport → one owning hand (press holder ≻ touching hand ≻ laser). Cursor dot per hand (shrinks as the tip nears = proximity cue), haptic tick on press + light ticks on hovering a new control, click release, slider steps (1/20 range), scroll detents (MenuScreen.Haptic), synthesized click (code-generated AudioStreamWav, no asset). Menu actions run DEFERRED (never from inside the injected event). "Recalibrate body" calls Recalibrate() on the first node in group "calibration" (disabled when absent — the menu never creates it; Calibration.cs provides it). Locomotion defers grab/teleport to the panel when a hand is near it (WorldPanel.IsBlocking). Laser lines are owned by Locomotion.UpdatePointer: for the menu it asks WorldPanel.TryGetLaserDistance (beam only while that hand is in laser mode, beyond LaserMinDistance 0.35 m); markers keep the raycast path. Regression: scenes/menu_spike.tscn "MENU: overall PASS". Grab deltas and turn pivots are computed in WORLD space (tracking-space math made two-hand turns orbit the map center — do not regress this).
• Control scheme is deliberate: decide what lives on controllers vs. what lives in menus before stacking bindings — controller button budget is scarce, menus carry the rest.

Order of work

1. Get VR working at all — done: OpenXR + WiVRN verified in-headset, flat preview fallback works without a headset.
2. Input foundation (partial) — grip/trigger Float polling + ax_button summon verified from engine source; the deliberate full action-map pass happens with the UI milestone. Hand tracking spike still TODO (XRHandTracker through WiVRN, see openxr_hand_tracking.rst).
3. Locomotion: anchor-grab mode — done, NEEDS FIXING (AnchorMoveMode.cs + teleport markers + spawn anchoring): two-hand turning not 2-anchor based, simultaneous two-hand movement weird, teleport + lasers broken (owner report 2026-10-01; experiments, not current issues). Fix-or-replace deferred — owner's candidate replacement is physics thumbstick movement if the body-physics track lands.
4. UI: first prop panel — done (WorldPanel.cs), then REPLACED (2026-10-02, owner: knobs "incredibly chonky") by the SubViewport menu screen (MenuScreen + MainMenu) — the one allowed dense screen. Awaiting owner playtest.
5. Box3D validation spike — done, PASS (drop+contact, velocity-chase vs wall, headless via scenes/physics_spike.tscn).
6. Physics hands & props sandbox — grab v5.3 = SOLVER-OWNED RIGID WELD (patched addon). Research first (see research/vr-grabbing.md, vr-two-handed-grab.md + its dated CORRECTION, godot-box3d-weld.md — sourced taxonomy, Box3D internals, the whole v5.0→5.2 history). The owner rejected every point-joint variant as "hanging on a winch" (ball orbits the palm, anvil surface drifts, palms clip) — a grab must lock the object's POSE relative to the hand, not one point + a rotator. The addon is now locally patched (source + recipe in .vendor/, originals in addons/godot-box3d/bin-stock/, Linux binary only): Godot's ConeTwistJoint3D maps to Box3D's b3WeldJoint, pre-corrected so the weld locks the relative pose AT CREATION (no identity-target snap; bias semantics INVERTED vs pin — 0 = max stiffness). PropGrabber.cs: grab creates a ConeTwistJoint3D weld at the palm point (SwingSpan/TwistSpan 0, ExcludeNodesFromCollision, configured BEFORE AddChild — mandatory ordering). NO script forces/torques/damp anywhere and NO per-tick joint re-positioning (the weld is not a hand-following pivot) — the solver owns everything and weight flows into the hand's 90 N budget honestly. BOTH hands weld RIGID (bias 0 — v5.3.1: the owner rejected the soft secondary; hand₁+prop+hand₂ welded = one composite body, conflicting controller demands resolve through hand lag, not object stretch); takeover is automatic — the survivor's weld persists, and the brief ~3° ring when one hand releases is stored weld stress discharging (honest physics). Break: stretch-based (rip 0.8 m / 0.25 m for 0.25 s / both-hands 0.4 m) + "grip dragged off" when the hand body lags its controller > PhysicsHands.SnapDistance. Headless regressions: scenes/weld_spike.tscn (addon patch), scenes/physics_spike.tscn, scenes/grab_spike.tscn (rigid pose under drag + fast spin — the winch regression; don't tighten rotation limits below ~2°, that's the gravity floor for off-center grabs). Grab v5.3.1 VALIDATED in-headset (owner: "it just works") — grab saga closed. PhysicsHands.cs: physical palm bodies (0.4 kg, 90 N chase, rotation lock, stuck-safety snap — SUSPENDED for a hand while it grips, HAND ghosts > 10 cm lag). Collision layers: 1 = world+props, 2 = UI, 3 = player body (PhysicsTorso chest), 4 = player hands, 5 = player arms (PhysicsArms segments) — no self-collision between own body parts (Nimso/VRChat precedent); props/floor/table mask 1|3|4|5, hands 4/1|4, chest 3/1|3, arm segments 5/1; panel stays 2/0 (props pass through even when shown); layers live on CollisionObject3D, NOT CollisionShape3D. Panel = the in-VR menu (ax_button summon, touch/laser Control UI, Reset props). Laser/distant grab REMOVED (user never knew it existed); grab = palm-proximity only, per-hand palm-face offset (left +X, right −X — taste). PinJoint quirks (anchors captured at creation; joint node position READ every tick; break measures the prop-local anchor vs the palm point) apply to the v5.0–5.2 PIN path only — the weld does not share the pivot behavior. Research reports live in research/ (nimso-project-tx.md = the Project TX player design VR Messs is recreating — read first for any body work, vr-physics-rig-hexabody.md = Boneworks/BIMOS reference + Box3D joint mapping, arm IK, IK stability/deep, grabbing, two-handed grab/Box3D internals, addon weld patch, body physics hugs, body physics Godot facts) — READ before re-researching or changing grab/IK/body. Stink-O's box3d-godot (custom components) = documented plan-B.
7. Body physics Stage 1 — PHYSICAL CHEST, implemented 2026-10-01, FAILED first playtest. PhysicsTorso.cs: one chase-driven RigidBody3D (30 kg box, gravity off, CCD) toward an HMD-derived target — the PhysicsHands PD pattern at body mass (2400 N budget), orientation = slewed angular-velocity servo (self-rights, holds yaw), 2 m snap safety, torso-yaw spring moved here from BodyRig. Camera stays KINEMATIC — torso chases the head, never the reverse (Boneworks sickness lesson). No self-collision with own hands (layer scheme above). First in-headset session NaN'd the chest — ROOT-CAUSED + FIXED same day: script math, not the solver (Quaternion.GetAngle() = 2·acos(W) → NaN at W = 1.0000001; see research/body-stage1-failure-brief.md RESOLUTION). Never use GetAngle()/GetAxis() for servo errors — BodyMath.RotationError (atan2) is the rule; every driven body runs BodyMath.ResetIfNonFinite. Regression: scenes/collapse_spike.tscn (run headless with --fixed-fps 90) must print "COLLAPSE: overall PASS". BodyRig.cs is VISUALS-only (head cube + arm boxes drawn through the physical elbow; IK + reach budget moved to PhysicsArms, item 10). Headless regression: scenes/body_spike.tscn must print "BODY: overall PASS" (also proves the Box3D pair rules: one-way contact, hand-through-chest exclusion, prop contact) — note its repro gap: no static-table overlap, no welded heavy props. Stage 2 (physical arm chains + hierarchical per-joint budgets, Nimso/Boneworks tier) — see item 10; design + prior art in research/vr-body-physics-hugs.md, stack facts in research/vr-body-physics-godot.md.
8. Getting a body (rig) — placeholder v1.3 (cube character, agent-built). (The IK/yaw/reach math described here now lives in PhysicsArms.cs / PhysicsTorso.cs; BodyRig.cs only draws.) Original notes: all-BOX primitives (owner preference) — head 0.15³ behind the eyes, slim torso retracted, wrist-anchored 2-bone arms. Soft IK applied: shoulder→hand distance clamped to reach×0.98 before the solve (the straight-arm singularity was the overextension shake; if it persists, lower SoftIKFactor toward 0.95). Elbow pole: torso frame, shoulder→hand axis PROJECTED OUT, hand weight 0.25, exp-smoothed 10/s (twitchy wrists fixed). Torso yaw spring: HMD yaw base blended toward hands-derived yaw (weight 0.6 × forwardness over 0.3 m), exp-smoothed 12/s, decoupled from the head — the "chest = headset + hands" ask. Reach budget hysteresis unchanged. Full sourced findings in research/vr-arm-ik.md and research/vr-arm-ik-deep.md (HL:Alyx went hands-only on purpose; SkeletonIK3D deprecated → custom SkeletonModifier3D when a real skeleton arrives; LookAtModifier3D for head→neck). EMBODYING (view-from-character-perspective) comes after the head/neck look override exists. HANDOFF.md at project root is the takeover document (constant table verified against code — code wins on drift); keep it updated when systems change.
9. Body rebuild toward Nimso's Project TX (research/nimso-project-tx.md) — step 0 DONE (fork 6-DOF joints, research/godot-box3d-6dof.md); step 1 DONE 2026-10-02: one-time T-pose CALIBRATION persisted in a gitignored prefs.json next to the executable (forced gate on launch, Recalibrate() via group "calibration"), BodyProfile (Winter anthropometry, arm stored 1.5 cm short), shared ArmSolver, and the TRUE BODY solver (TrueBody.cs: neck, chest yaw + crouch lean, lagging pelvis, moving shoulder girdle, arms) with a debug skeleton. Arm-length decisions: research/vr-avatar-arm-length.md. Step 2 DONE 2026-10-02 (awaiting owner playtest): the player IS a muscle-driven ragdoll — PlayerRagdoll.cs (9 bodies, BALL/hinge/ DRIVE joints with torque-capped motor PDs toward the True Body, chest→hand guides, implicit balance, ground control) + PlayerRig.cs (XR driver: stick move/snap turn, Bonelab push-back via an eye→camera sphere sweep, discrete origin jumps carry the body). It REPLACED PhysicsHands/PhysicsTorso/PhysicsArms/BodyRig in main.tscn (legacy files kept for their spikes). Record + measured rules: research/ragdoll-player-step2.md — never put a light (< ~1 kg) body between two strong motors (Box3D one-iteration substeps oscillate). Regression scenes/ragdoll_spike.tscn. Next: procedural legs, 144 Hz, climbing. Owner scope: thumbstick locomotion only, no jump, avatar scaling and NPCs later.
10. Body physics Stage 2 — PHYSICAL ARMS, implemented 2026-10-02, awaiting owner playtest. PhysicsArms.cs: per side an upper-arm + forearm RigidBody3D (2.0 / 1.2 kg, layer 5) that PD-chase their analytic 2-bone IK pose, solved each physics tick from the PHYSICAL chest shoulders to the PHYSICAL hands. No joints — Box3D has no usable 3-DOF shoulder/wrist joint (PinJoint's hidden identity spring, no Generic6DOF) and this file's architecture forbids joints between tracked bodies; the research's hinge-motor chain is therefore deferred, chase segments are the Box3D-honest version. Hierarchical budgets (Nimso muscle model) from a 60 kg body-mass stat: hand 90 N ≺ forearm 180 N ≺ upper arm 360 N ≺ chest 2400 N. What it gives: arms that push props, rest on / get blocked by geometry (no tunneling), and draw honestly (BodyRig's elbow = the physical elbow). Playtest fixes (2026-10-02): open-loop torso lean (no chest hunting), controller-driven + clamped/rate-limited elbow (no flips), held-prop arm exceptions, and Stage 2b HAND COUPLING (a blocked arm holds its hand back, 3 cm slack, ghost shows the controller). Still NOT done: prop/NPC forces on the arm don't flow into the chest. Regression: scenes/arm_spike.tscn "ARM: overall PASS"; the full body also runs inside scenes/collapse_spike.tscn. Design + prior art: research/vr-body-physics-hugs.md §5–7, research/vr-body-physics-godot.md.