VR Messs

VR anime-character "scene viewer" (Koikatsu-VR-class, but with real physics hands). Not a game: no gameplay loops, quests, or progression. Gameplay features are the owner's domain; explore, interact, touch, pose — that's the product.

TERMINOLOGY (owner-corrected, use precisely): "getting a body" = the physical rig (torso/arms/hands — the interaction body). "embodying" = literally viewing from the perspective of a character in an animation (the view-follow + avatar system). They are separate systems; never call the rig milestone "embodying".

This repo is the experimentation lab (hence the name). The designs below are working hypotheses to be validated with cheap, throwaway spikes — a failed experiment is progress, not rework. The embodiment/view system in particular is theory until proven.

Toolchain pins (do not drift)

Godot 4.7.2 (current stable, Forward Plus renderer), C# / .NET 8. Desktop-only (PCVR).
Physics: Box3D via the godot-box3d GDExtension (server-replacement package by bearlikelion) — installed, v0.2.6, setting value physics/3d/physics_engine="Box3D Physics". Validation spike (scenes/physics_spike.tscn, run headless) PASSES: drop+contact and velocity-chase-into-wall both work from C#. WARNING: exactly ONE .gdextension file may exist — a stray duplicate (e.g. at addons/ root) double-loads the library and breaks the server with "required virtual method" errors. Jolt (built-in) is the fallback: reverting is the one project-setting line, and all gameplay code must stay standard-Godot-physics-node based so that flip is lossless.
Character import (Koikatsu etc.) is a separate pipeline owned by someone else. We consume a character scene that provides: Skeleton3D with humanoid bone mapping + AnimationPlayer. Don't touch import code; if the interface is insufficient, say so instead of hacking around it.
Dev runtime stack: WiVRN + WayVR on Linux. Headset and runtime both support hand tracking — treat hand tracking as available. View-follow milestone verified working (OpenXR + WiVRN).
Graphics are deliberately stylized — fits the anime look and buys performance headroom. Renderer choice (currently Forward+; docs recommend Mobile for desktop VR) is deferred until profiling justifies a change.

Documentation rules (hard rules — this project's #1 failure mode)

Local pinned docs: /home/foxyz/Projects/Games/.shared/godot-docs-4.7 — full manual + class reference in .rst (source of docs.godotengine.org, branch 4.7). Grep it first.
For ANY engine API detail, class name, property, or signal: trust only these 4.7 docs or a live spike. Training-memory APIs and older-version knowledge are banned — verify or don't write it.
Web fallback: https://docs.godotengine.org/en/4.7/... only — the version goes IN the URL. Never /latest/, never another version path.
Code examples: rst sources contain code-tab:: blocks with BOTH gdscript and csharp — grep for code-tab:: csharp. Class reference lives in classes/class_*.rst.
XR tutorials: tutorials/xr/ (setting_up_xr, xr_action_map, xr_room_scale, openxr_body_tracking, basic_xr_locomotion). Physics: tutorials/physics/ (rigid_body, ragdoll_system, using_jolt_physics, physics_introduction).

Godot/C# traps

Addons that are GDScript-only can't be referenced from C# — Godot XR Tools is GDScript-only: treat it as a reference implementation to port ideas from, never a dependency.
Default OpenXR action map names (verified from engine source, 4.7): trigger/grip (Float), ax_button (A/X), by_button (B/Y), menu_button (menu/system — often eaten by the runtime, e.g. WiVRN screenshot+anchor-hold), primary (thumbstick). select_button is Pico-only. Poses: default_pose, aim_pose, grip_pose, palm_pose.
C# in Godot covers desktop + mobile only (fine — PCVR target).
Signals: C# +=/-= events; check the class ref for exact C# signatures.
The C# binding can diverge from engine class names: AudioStreamWAV is AudioStreamWav in C#. When docs and compiler disagree, grep the shipped GodotSharp.dll (strings) — it's at /usr/lib/godot-mono/GodotSharp/Api/Debug/GodotSharp.dll on this machine.

Architecture (working hypotheses — test cheap before committing)

Four layers, bottom-up, applied per frame in this order:

Animation — AnimationPlayer drives the skeleton (any pose, any animation).
Tracked override / view embodiment — several input modes, all blended over whatever animation is playing:
- View-follow (core feature, Koikatsu-VR-style): the view follows the character's head frame including tilt, so weird animations stay correct. The player's HMD look is remapped through a one-time calibration transform captured relative to the character's chest ("chest calibration", e.g. once while lying flat) and drives a look override on the head+neck (possibly slight chest) bones. Skeleton-space only — the physics layer stays separate from this. Pivot behavior (rotation-only snap vs positional head offset) and the calibration UX are open experiments.
- Free-view: ordinary playspace (XROrigin3D); "lean back mode" = origin offset + recline calibration, zero physics involved.
- Single-controller mode: one controller as master tracker — hand override, or strapped to the hip driving the animation root/hip (hip-tracker mode).
- Full-body trackers are a far-future extension of this same layer (more override targets), not a separate system.
Physics embodiment — a small set of rigid bodies (hand L/R, head, torso/hip capsule) that CHASE layer-1/2 targets: velocity-clamped steering (v = clamp((target - pos) / dt)), CCD on, no joints between them. Unobstructed → effectively 1:1 tracking; obstructed → pushes with tunable strength. The chase deviation is the feedback signal: it clamps the visual hand offset, pushes the character pose via spring-damped offsets ("pushable from animation"), and fires logical touch-reaction events. Boneworks calls the general technique "heuristic force limit".
Secondary motion (hair/skirt/cloth) — custom spring-bone solver. Koikatsu DynamicBones and VRM springBone are poorly interactive and/or expensive; VRChat PhysBones are the gold standard but proprietary. So: original PhysBones-style math (spring-damper chains on the skeleton, animation-space) with analytic sphere/capsule colliders (body, hands, floor). The physics engine contributes only its QUERY API (throttled ray/shape casts) so bones respect scene geometry — never rigid-body chains for jiggle. Performance is a first-class goal of this project: chain budgets, screen-size/distance LOD, simplified colliders. If the per-bone C# loop profiles hot, the hot loop moves to a GDExtension — profile first, don't pre-optimize.

Do NOT build anything that needs 6DOF/cone-twist joints between tracked bodies (the wrapper lacks them; velocity steering replaces them). Scene objects stay standard nodes (RigidBody3D, StaticBody3D, CharacterBody3D) so arbitrary imported scene content works and the Jolt fallback stays real.

XR

OpenXR is built into Godot 4.7 — no plugin. Node set: XROrigin3D / XRCamera3D / XRController3D (+ XRBodyTracker/XRHandTracker for body/hand tracking data). OpenXR runs on device only; the editor has an XR simulator for dev. Trackers/action map: see xr_action_map.rst.

Interaction & UI (design directions — experiments welcome)

Touch-first hybrid cursor: interactive surfaces prefer touch. A pointer tip (controller tip or index fingertip) acts as a cursor when in proximity of a surface — push, hold, slide, physical press feel. Out of proximity, fall back to a laser ray for distant precision. Lasers ruin immersion and are hard to aim at distance; they are the fallback, not the default.
Hand tracking: cheap spike early, full support later. Interaction logic must not assume controllers — controller tip and fingertip feed the same cursor concept.
Locomotion (implemented, play-feel tuning ongoing): spawn re-anchoring happens ONCE on first focus gained (HMD xz at SpawnPoint + spawn yaw; real head height and pitch kept) — physical room position never matters. pose_recentered is deliberately NOT a placement trigger AND its log is throttled to one line: measured on WiVRN it fires continuously at refresh rate (~90 Hz, 22k lines/session) — reacting to it was the original view-locked-forward bug. Locomotion.cs is the manager (spawn placement, teleport markers, lasers, fade); grab movement lives in AnchorMoveMode.cs as its own movement mode — scene-viewer-first, other modes (walk, tp-only variants) are future nodes beside it. Teleport markers (scenes/teleport_marker.tscn): aim + trigger, with fade; near markers (within MarkerNearDistance 1.2 m) fully hide (disc, beam, label) AND are not teleport-targetable. Grab semantics are ANCHOR-based: hands are anchors to push/pull from — one-hand grip moves the player OPPOSITE to hand motion in full 3D, both grips = anchor translation + IN-PLACE yaw turn around the head. All grab deltas and turn pivots are computed in WORLD space (tracking-space math made turns orbit the map center — do not regress). Spawn/teleport placement pins arrival height (head height preserved). Locomotion drives XROrigin3D directly; physics reconciliation comes with the embodiment layer. Grip/trigger are polled Floats with thresholds per docs (bool actions unreliable across runtimes). Engine logs land in ~/.local/share/godot/app_userdata/VR Messs/logs/godot.log — agents: read this after user playtests instead of guessing.
UI direction — physical prop controls first: buttons, knobs, sliders as real 3D objects (touch interaction, later Box3D for feel). Godot has NO native world-space widget system — Controls are screen-space; world UI is always Control-rendered-into-3D via SubViewport (optionally SubViewportCompositionLayer for XR-native sharpness/perf). Use at most one SubViewport/composition-layer screen for dense content (file browser, settings) if needed; everything else is props. World-space text is native and cheap: Label3D (SDF, crisp at distance) and TextMesh (extruded) — no SubViewport required.
WorldPanel.cs is the first prop-UI prototype: summoned/dismissed via ax_button (A/X — menu_button belongs to the runtime). It is PARENTED UNDER XROrigin3D — Koikatsu-VR-style "fixed in the player's room": IRL movement doesn't move it relative to you, game locomotion carries it. If that feel is wrong, reverting = reparenting to Main and fixing the _Ready node paths. Floats in front of the summoning hand facing the player (full 3D facing — works from lying poses), draggable directly or via laser at fixed ray distance while gripped (continuously re-orients toward the head). The Spawn knob also recalls the panel in front of the head. The pointer tip is a protruding stylus cone (apex ~12 cm ahead of the controller) and is THE cursor: knobs hover-highlight and PRESS ON CONTACT (tip within 4.5 cm, with cooldown) — physical buttons, no trigger needed; trigger still presses laser-hovered knobs (knobs are laser-targetable). Presses play a synthesized click (code-generated AudioStreamWav PCM — no asset). Spawn/Ping/Close knobs. Locomotion defers grab/teleport to the panel when a hand is near it (WorldPanel.IsBlocking). Laser lines are owned by Locomotion.UpdatePointer: shown only when aiming at something interactive (marker or panel) AND beyond LaserMinDistance (0.35 m) — near targets are touched via the pointer tip instead, which is the touch-first hybrid cursor in miniature. Grab deltas and turn pivots are computed in WORLD space (tracking-space math made two-hand turns orbit the map center — do not regress this).
Control scheme is deliberate: decide what lives on controllers vs. what lives in menus before stacking bindings — controller button budget is scarce, menus carry the rest.

Order of work

~~Get VR working at all~~ — done: OpenXR + WiVRN verified in-headset, flat preview fallback works without a headset.
~~Input foundation (partial)~~ — grip/trigger Float polling + ax_button summon verified from engine source; the deliberate full action-map pass happens with the UI milestone. Hand tracking spike still TODO (XRHandTracker through WiVRN, see openxr_hand_tracking.rst).
~~Locomotion: anchor-grab mode~~ — done (AnchorMoveMode.cs + teleport markers + spawn anchoring); other movement modes are future nodes beside it.
~~UI: first prop panel~~ — done (WorldPanel.cs); SubViewport/composition-layer screen deferred until dense content (file browser) is actually needed.
~~Box3D validation spike~~ — done, PASS (drop+contact, velocity-chase vs wall, headless via scenes/physics_spike.tscn).
Physics hands & props sandbox — grab v5.3 = SOLVER-OWNED RIGID WELD (patched addon). Research first (see research/vr-grabbing.md, vr-two-handed-grab.md + its dated CORRECTION, godot-box3d-weld.md — sourced taxonomy, Box3D internals, the whole v5.0→5.2 history). The owner rejected every point-joint variant as "hanging on a winch" (ball orbits the palm, anvil surface drifts, palms clip) — a grab must lock the object's POSE relative to the hand, not one point + a rotator. The addon is now locally patched (source + recipe in .vendor/, originals in addons/godot-box3d/bin-stock/, Linux binary only): Godot's ConeTwistJoint3D maps to Box3D's b3WeldJoint, pre-corrected so the weld locks the relative pose AT CREATION (no identity-target snap; bias semantics INVERTED vs pin — 0 = max stiffness). PropGrabber.cs: grab creates a ConeTwistJoint3D weld at the palm point (SwingSpan/TwistSpan 0, ExcludeNodesFromCollision, configured BEFORE AddChild — mandatory ordering). NO script forces/torques/damp anywhere and NO per-tick joint re-positioning (the weld is not a hand-following pivot) — the solver owns everything and weight flows into the hand's 90 N budget honestly. BOTH hands weld RIGID (bias 0 — v5.3.1: the owner rejected the soft secondary; hand₁+prop+hand₂ welded = one composite body, conflicting controller demands resolve through hand lag, not object stretch); takeover is automatic — the survivor's weld persists, and the brief ~3° ring when one hand releases is stored weld stress discharging (honest physics). Break: stretch-based (rip 0.8 m / 0.25 m for 0.25 s / both-hands 0.4 m) + "grip dragged off" when the hand body lags its controller > PhysicsHands.SnapDistance. Headless regressions: scenes/weld_spike.tscn (addon patch), scenes/physics_spike.tscn, scenes/grab_spike.tscn (rigid pose under drag + fast spin — the winch regression; don't tighten rotation limits below ~2°, that's the gravity floor for off-center grabs). PhysicsHands.cs: physical palm bodies (0.4 kg, 90 N chase, rotation lock, stuck-safety snap — SUSPENDED for a hand while it grips, HAND ghosts > 10 cm lag). Collision layers: 1 = world, 2 = UI (panel mask 0 — props pass through even when shown); layers live on CollisionObject3D, NOT CollisionShape3D. Panel = prop-UI prototype (ax_button summon, contact-press knobs, Reset). Research reports live in research/ (arm IK, IK stability/deep, grabbing, two-handed grab/Box3D internals, addon weld patch) — READ before re-researching or changing grab/IK. Stink-O's box3d-godot (custom components) = documented plan-B. Laser/distant grab REMOVED (user never knew it existed); grab = palm-proximity only, per-hand palm-face offset (left +X, right −X — taste). Box3D PinJoint quirks: anchors are CAPTURED AT CREATION; the joint node's position is READ EVERY TICK (keep it on the palm = hand-following pivot); break must measure the prop-local anchor vs the palm point. Headless regression tests: scenes/physics_spike.tscn AND scenes/grab_spike.tscn — both must print overall PASS. PhysicsHands.cs: physical palm bodies (0.4 kg, 90 N chase, rotation lock, stuck-safety snap, HAND ghosts > 10 cm lag). Collision layers: 1 = world, 2 = UI (panel mask 0 — props pass through even when shown); layers live on CollisionObject3D, NOT CollisionShape3D. Panel = prop-UI prototype (ax_button summon, contact-press knobs, Reset). Research reports live in research/ (arm IK, IK stability/deep, grabbing, two-handed grab/Box3D joint internals) — READ before re-researching or changing grab/IK.
Getting a body (rig) — placeholder v1.2 (cube character, agent-built). BodyRig.cs: all-BOX primitives (owner preference) — head 0.15³ behind the eyes, slim torso retracted, wrist-anchored 2-bone arms. Soft IK applied: shoulder→hand distance clamped to reach×0.98 before the solve (the straight-arm singularity was the overextension shake; if it persists, lower SoftIKFactor toward 0.95). Elbow pole: torso frame, shoulder→hand axis PROJECTED OUT, hand weight 0.25, exp-smoothed 10/s (twitchy wrists fixed). Torso yaw spring: HMD yaw base blended toward hands-derived yaw (weight 0.6 × forwardness over 0.3 m), exp-smoothed 12/s, decoupled from the head — the "chest = headset + hands" ask. Reach budget hysteresis unchanged. Full sourced findings in research/vr-arm-ik.md and research/vr-arm-ik-deep.md (HL:Alyx went hands-only on purpose; SkeletonIK3D deprecated → custom SkeletonModifier3D when a real skeleton arrives; LookAtModifier3D for head→neck). Physical torso/head capsules later. EMBODYING (view-from-character-perspective) comes after the head/neck look override exists. HANDOFF.md at project root is the takeover document (constant table verified against code — code wins on drift); keep it updated when systems change.

18 KiB Raw Blame History Unescape Escape