237 lines
18 KiB
Markdown
237 lines
18 KiB
Markdown
# VR Messs
|
||
|
||
VR anime-character "scene viewer" (Koikatsu-VR-class, but with real physics hands).
|
||
Not a game: no gameplay loops, quests, or progression. Gameplay features are the
|
||
owner's domain; explore, interact, touch, pose — that's the product.
|
||
|
||
TERMINOLOGY (owner-corrected, use precisely): **"getting a body"** = the physical rig
|
||
(torso/arms/hands — the interaction body). **"embodying"** = literally viewing from the
|
||
perspective of a character in an animation (the view-follow + avatar system). They are
|
||
separate systems; never call the rig milestone "embodying".
|
||
|
||
This repo is the experimentation lab (hence the name). The designs below are working hypotheses
|
||
to be validated with cheap, throwaway spikes — a failed experiment is progress, not rework. The
|
||
embodiment/view system in particular is theory until proven.
|
||
|
||
## Toolchain pins (do not drift)
|
||
|
||
- **Godot 4.7.2** (current stable, Forward Plus renderer), **C# / .NET 8**. Desktop-only (PCVR).
|
||
- **Physics: Box3D** via the `godot-box3d` GDExtension (server-replacement package by
|
||
bearlikelion) — **installed, v0.2.6**, setting value `physics/3d/physics_engine="Box3D
|
||
Physics"`. Validation spike (`scenes/physics_spike.tscn`, run headless) PASSES: drop+contact
|
||
and velocity-chase-into-wall both work from C#. WARNING: exactly ONE `.gdextension` file may
|
||
exist — a stray duplicate (e.g. at `addons/` root) double-loads the library and breaks the
|
||
server with "required virtual method" errors. Jolt (built-in) is the fallback: reverting is
|
||
the one project-setting line, and all gameplay code must stay standard-Godot-physics-node
|
||
based so that flip is lossless.
|
||
- Character import (Koikatsu etc.) is a separate pipeline owned by someone else. We consume a
|
||
character scene that provides: `Skeleton3D` with humanoid bone mapping + `AnimationPlayer`.
|
||
Don't touch import code; if the interface is insufficient, say so instead of hacking around it.
|
||
- Dev runtime stack: **WiVRN + WayVR** on Linux. Headset and runtime both support hand tracking —
|
||
treat hand tracking as available. View-follow milestone verified working (OpenXR + WiVRN).
|
||
- Graphics are deliberately **stylized** — fits the anime look and buys performance headroom.
|
||
Renderer choice (currently Forward+; docs recommend Mobile for desktop VR) is deferred until
|
||
profiling justifies a change.
|
||
|
||
## Documentation rules (hard rules — this project's #1 failure mode)
|
||
|
||
- Local pinned docs: `/home/foxyz/Projects/Games/.shared/godot-docs-4.7` — full manual +
|
||
class reference in `.rst` (source of docs.godotengine.org, branch `4.7`). Grep it first.
|
||
- For ANY engine API detail, class name, property, or signal: **trust only these 4.7 docs or a
|
||
live spike.** Training-memory APIs and older-version knowledge are banned — verify or don't write it.
|
||
- Web fallback: `https://docs.godotengine.org/en/4.7/...` only — the version goes IN the URL.
|
||
Never `/latest/`, never another version path.
|
||
- Code examples: rst sources contain `code-tab::` blocks with BOTH gdscript and csharp — grep for
|
||
`code-tab:: csharp`. Class reference lives in `classes/class_*.rst`.
|
||
- XR tutorials: `tutorials/xr/` (setting_up_xr, xr_action_map, xr_room_scale,
|
||
openxr_body_tracking, basic_xr_locomotion). Physics: `tutorials/physics/` (rigid_body,
|
||
ragdoll_system, using_jolt_physics, physics_introduction).
|
||
|
||
## Godot/C# traps
|
||
|
||
- Addons that are GDScript-only can't be referenced from C# — **Godot XR Tools is GDScript-only**:
|
||
treat it as a reference implementation to port ideas from, never a dependency.
|
||
- Default OpenXR action map names (verified from engine source, 4.7): `trigger`/`grip` (Float),
|
||
`ax_button` (A/X), `by_button` (B/Y), `menu_button` (menu/system — often eaten by the runtime,
|
||
e.g. WiVRN screenshot+anchor-hold), `primary` (thumbstick). `select_button` is Pico-only.
|
||
Poses: `default_pose`, `aim_pose`, `grip_pose`, `palm_pose`.
|
||
- C# in Godot covers desktop + mobile only (fine — PCVR target).
|
||
- Signals: C# `+=`/`-=` events; check the class ref for exact C# signatures.
|
||
- The C# binding can diverge from engine class names: `AudioStreamWAV` is `AudioStreamWav`
|
||
in C#. When docs and compiler disagree, grep the shipped `GodotSharp.dll` (strings) —
|
||
it's at `/usr/lib/godot-mono/GodotSharp/Api/Debug/GodotSharp.dll` on this machine.
|
||
|
||
## Architecture (working hypotheses — test cheap before committing)
|
||
|
||
Four layers, bottom-up, applied per frame in this order:
|
||
|
||
1. **Animation** — `AnimationPlayer` drives the skeleton (any pose, any animation).
|
||
2. **Tracked override / view embodiment** — several input modes, all blended over whatever
|
||
animation is playing:
|
||
- **View-follow (core feature, Koikatsu-VR-style)**: the view follows the character's head
|
||
frame including tilt, so weird animations stay correct. The player's HMD look is remapped
|
||
through a one-time calibration transform captured relative to the character's chest
|
||
("chest calibration", e.g. once while lying flat) and drives a look override on the
|
||
head+neck (possibly slight chest) bones. Skeleton-space only — the physics layer stays
|
||
separate from this. Pivot behavior (rotation-only snap vs positional head offset) and the
|
||
calibration UX are open experiments.
|
||
- **Free-view**: ordinary playspace (`XROrigin3D`); "lean back mode" = origin offset +
|
||
recline calibration, zero physics involved.
|
||
- **Single-controller mode**: one controller as master tracker — hand override, or strapped
|
||
to the hip driving the animation root/hip (hip-tracker mode).
|
||
- Full-body trackers are a far-future extension of this same layer (more override targets),
|
||
not a separate system.
|
||
3. **Physics embodiment** — a small set of rigid bodies (hand L/R, head, torso/hip capsule) that
|
||
CHASE layer-1/2 targets: velocity-clamped steering (`v = clamp((target - pos) / dt)`), CCD on,
|
||
no joints between them. Unobstructed → effectively 1:1 tracking; obstructed → pushes with
|
||
tunable strength. The chase deviation is the feedback signal: it clamps the visual hand offset,
|
||
pushes the character pose via spring-damped offsets ("pushable from animation"), and fires
|
||
logical touch-reaction events. Boneworks calls the general technique "heuristic force limit".
|
||
4. **Secondary motion (hair/skirt/cloth) — custom spring-bone solver.** Koikatsu DynamicBones and
|
||
VRM springBone are poorly interactive and/or expensive; VRChat PhysBones are the gold standard
|
||
but proprietary. So: original PhysBones-*style* math (spring-damper chains on the skeleton,
|
||
animation-space) with analytic sphere/capsule colliders (body, hands, floor). The physics
|
||
engine contributes only its QUERY API (throttled ray/shape casts) so bones respect scene
|
||
geometry — **never rigid-body chains for jiggle**. Performance is a first-class goal of this
|
||
project: chain budgets, screen-size/distance LOD, simplified colliders. If the per-bone C#
|
||
loop profiles hot, the hot loop moves to a GDExtension — profile first, don't pre-optimize.
|
||
|
||
Do NOT build anything that needs 6DOF/cone-twist joints between tracked bodies (the wrapper lacks
|
||
them; velocity steering replaces them). Scene objects stay standard nodes (`RigidBody3D`,
|
||
`StaticBody3D`, `CharacterBody3D`) so arbitrary imported scene content works and the Jolt fallback
|
||
stays real.
|
||
|
||
## XR
|
||
|
||
OpenXR is built into Godot 4.7 — no plugin. Node set: `XROrigin3D` / `XRCamera3D` /
|
||
`XRController3D` (+ `XRBodyTracker`/`XRHandTracker` for body/hand tracking data). OpenXR runs on
|
||
device only; the editor has an XR simulator for dev. Trackers/action map: see `xr_action_map.rst`.
|
||
|
||
## Interaction & UI (design directions — experiments welcome)
|
||
|
||
- **Touch-first hybrid cursor**: interactive surfaces prefer *touch*. A pointer tip (controller
|
||
tip or index fingertip) acts as a cursor when in proximity of a surface — push, hold, slide,
|
||
physical press feel. Out of proximity, fall back to a laser ray for distant precision. Lasers
|
||
ruin immersion and are hard to aim at distance; they are the fallback, not the default.
|
||
- **Hand tracking**: cheap spike early, full support later. Interaction logic must not assume
|
||
controllers — controller tip and fingertip feed the same cursor concept.
|
||
- **Locomotion** (implemented, play-feel tuning ongoing): spawn re-anchoring
|
||
happens ONCE on first focus gained (HMD xz at `SpawnPoint` + spawn yaw; real head height and
|
||
pitch kept) — physical room position never matters. `pose_recentered` is deliberately NOT a
|
||
placement trigger AND its log is throttled to one line: measured on WiVRN it fires
|
||
continuously at refresh rate (~90 Hz, 22k lines/session) — reacting to it was the original
|
||
view-locked-forward bug. `Locomotion.cs` is the manager (spawn placement,
|
||
teleport markers, lasers, fade); **grab movement lives in `AnchorMoveMode.cs` as its own
|
||
movement mode** — scene-viewer-first, other modes (walk, tp-only variants) are future nodes
|
||
beside it. Teleport markers (`scenes/teleport_marker.tscn`):
|
||
aim + trigger, with fade; near markers (within `MarkerNearDistance` 1.2 m) fully hide
|
||
(disc, beam, label) AND are not teleport-targetable. Grab semantics are
|
||
ANCHOR-based: hands are anchors to push/pull from — one-hand grip
|
||
moves the player OPPOSITE to hand motion in full 3D, both grips = anchor translation +
|
||
IN-PLACE yaw turn around the head. All grab deltas and turn pivots are computed in
|
||
WORLD space (tracking-space math made turns orbit the map center — do not regress).
|
||
Spawn/teleport placement pins arrival height (head height preserved). Locomotion drives
|
||
`XROrigin3D` directly; physics reconciliation comes with the embodiment layer. Grip/trigger
|
||
are polled Floats with thresholds per docs
|
||
(bool actions unreliable across runtimes). Engine logs land in
|
||
`~/.local/share/godot/app_userdata/VR Messs/logs/godot.log` — agents: read this after user
|
||
playtests instead of guessing.
|
||
- **UI direction — physical prop controls first**: buttons, knobs, sliders as real 3D objects
|
||
(touch interaction, later Box3D for feel). Godot has NO native world-space widget system —
|
||
`Control`s are screen-space; world UI is always Control-rendered-into-3D via `SubViewport`
|
||
(optionally `SubViewportCompositionLayer` for XR-native sharpness/perf). Use at most one
|
||
SubViewport/composition-layer screen for dense content (file browser, settings) if needed;
|
||
everything else is props. World-space text is native and cheap: `Label3D` (SDF, crisp at
|
||
distance) and `TextMesh` (extruded) — no SubViewport required.
|
||
- **`WorldPanel.cs` is the first prop-UI prototype**: summoned/dismissed via `ax_button` (A/X —
|
||
menu_button belongs to the runtime). It is PARENTED UNDER `XROrigin3D` — Koikatsu-VR-style
|
||
"fixed in the player's room": IRL movement doesn't move it relative to you, game locomotion
|
||
carries it. If that feel is wrong, reverting = reparenting to `Main` and fixing the `_Ready`
|
||
node paths. Floats in front of the summoning hand facing the player (full 3D facing — works
|
||
from lying poses), draggable directly or via laser at fixed ray distance while gripped
|
||
(continuously re-orients toward the head). The Spawn
|
||
knob also recalls the panel in front of the head. The pointer
|
||
tip is a protruding stylus cone (apex ~12 cm ahead of the controller) and is THE cursor:
|
||
knobs hover-highlight and PRESS ON CONTACT (tip within 4.5 cm, with cooldown) — physical
|
||
buttons, no trigger needed; trigger still presses laser-hovered knobs (knobs are
|
||
laser-targetable). Presses play a synthesized click (code-generated `AudioStreamWav` PCM
|
||
— no asset). Spawn/Ping/Close knobs. Locomotion defers
|
||
grab/teleport to the panel when a hand is near it (`WorldPanel.IsBlocking`). Laser lines are
|
||
owned by `Locomotion.UpdatePointer`: shown only when aiming at something interactive (marker
|
||
or panel) AND beyond `LaserMinDistance` (0.35 m) — near targets are touched via the pointer
|
||
tip instead, which is the touch-first hybrid cursor in miniature. Grab deltas and turn
|
||
pivots are computed in WORLD space (tracking-space math made two-hand turns orbit the map
|
||
center — do not regress this).
|
||
- **Control scheme is deliberate**: decide what lives on controllers vs. what lives in menus
|
||
before stacking bindings — controller button budget is scarce, menus carry the rest.
|
||
|
||
## Order of work
|
||
|
||
1. ~~Get VR working at all~~ — **done**: OpenXR + WiVRN verified in-headset, flat preview
|
||
fallback works without a headset.
|
||
2. ~~Input foundation (partial)~~ — grip/trigger Float polling + `ax_button` summon verified
|
||
from engine source; the deliberate full action-map pass happens with the UI milestone.
|
||
**Hand tracking spike still TODO** (`XRHandTracker` through WiVRN, see
|
||
`openxr_hand_tracking.rst`).
|
||
3. ~~Locomotion: anchor-grab mode~~ — **done** (`AnchorMoveMode.cs` + teleport markers +
|
||
spawn anchoring); other movement modes are future nodes beside it.
|
||
4. ~~UI: first prop panel~~ — **done** (`WorldPanel.cs`); SubViewport/composition-layer screen
|
||
deferred until dense content (file browser) is actually needed.
|
||
5. ~~Box3D validation spike~~ — **done, PASS** (drop+contact, velocity-chase vs wall, headless
|
||
via `scenes/physics_spike.tscn`).
|
||
6. **Physics hands & props sandbox — grab v5.3 = SOLVER-OWNED RIGID WELD (patched addon).**
|
||
Research first (see `research/vr-grabbing.md`, `vr-two-handed-grab.md` + its dated
|
||
CORRECTION, `godot-box3d-weld.md` — sourced taxonomy, Box3D internals, the whole v5.0→5.2
|
||
history). The owner rejected every point-joint variant as "hanging on a winch" (ball
|
||
orbits the palm, anvil surface drifts, palms clip) — a grab must lock the object's POSE
|
||
relative to the hand, not one point + a rotator. **The addon is now locally patched**
|
||
(source + recipe in `.vendor/`, originals in `addons/godot-box3d/bin-stock/`, Linux binary
|
||
only): Godot's ConeTwistJoint3D maps to Box3D's `b3WeldJoint`, pre-corrected so the weld
|
||
locks the relative pose AT CREATION (no identity-target snap; bias semantics INVERTED vs
|
||
pin — 0 = max stiffness). `PropGrabber.cs`: grab creates a ConeTwistJoint3D weld at the
|
||
palm point (`SwingSpan/TwistSpan 0`, `ExcludeNodesFromCollision`, configured BEFORE
|
||
AddChild — mandatory ordering). NO script forces/torques/damp anywhere and NO per-tick
|
||
joint re-positioning (the weld is not a hand-following pivot) — the solver owns everything
|
||
and weight flows into the hand's 90 N budget honestly. BOTH hands weld RIGID (bias 0 —
|
||
v5.3.1: the owner rejected the soft secondary; hand₁+prop+hand₂ welded = one composite
|
||
body, conflicting controller demands resolve through hand lag, not object stretch);
|
||
takeover is automatic — the survivor's weld persists, and the brief ~3° ring when one
|
||
hand releases is stored weld stress discharging (honest physics). Break: stretch-based (rip 0.8 m / 0.25 m for 0.25 s / both-hands
|
||
0.4 m) + "grip dragged off" when the hand body lags its controller > `PhysicsHands.SnapDistance`.
|
||
Headless regressions: `scenes/weld_spike.tscn` (addon patch), `scenes/physics_spike.tscn`,
|
||
`scenes/grab_spike.tscn` (rigid pose under drag + fast spin — the winch regression; don't
|
||
tighten rotation limits below ~2°, that's the gravity floor for off-center grabs).
|
||
`PhysicsHands.cs`: physical palm bodies (0.4 kg, 90 N chase, rotation lock, stuck-safety
|
||
snap — SUSPENDED for a hand while it grips, HAND ghosts > 10 cm lag). Collision layers:
|
||
1 = world, 2 = UI (panel mask 0 — props pass through even when shown); layers live on
|
||
CollisionObject3D, NOT CollisionShape3D. Panel = prop-UI prototype (ax_button summon,
|
||
contact-press knobs, Reset). Research reports live in `research/` (arm IK, IK
|
||
stability/deep, grabbing, two-handed grab/Box3D internals, addon weld patch) — READ before
|
||
re-researching or changing grab/IK. Stink-O's box3d-godot (custom components) = documented
|
||
plan-B. Laser/distant grab REMOVED (user never knew it existed); grab = palm-proximity only,
|
||
per-hand palm-face offset (left +X, right −X — taste). Box3D PinJoint quirks: anchors are
|
||
CAPTURED AT CREATION; the joint node's position is READ EVERY TICK (keep it on the palm =
|
||
hand-following pivot); break must measure the prop-local anchor vs the palm point. Headless
|
||
regression tests: `scenes/physics_spike.tscn` AND `scenes/grab_spike.tscn` — both must print
|
||
overall PASS. `PhysicsHands.cs`: physical palm bodies (0.4 kg, 90 N chase, rotation lock,
|
||
stuck-safety snap, HAND ghosts > 10 cm lag). Collision layers: 1 = world, 2 = UI (panel mask
|
||
0 — props pass through even when shown); layers live on CollisionObject3D, NOT
|
||
CollisionShape3D. Panel = prop-UI prototype (ax_button summon, contact-press knobs, Reset).
|
||
Research reports live in `research/` (arm IK, IK stability/deep, grabbing,
|
||
two-handed grab/Box3D joint internals) — READ before re-researching or changing grab/IK.
|
||
7. **Getting a body (rig) — placeholder v1.2 (cube character, agent-built).** `BodyRig.cs`:
|
||
all-BOX primitives (owner preference) — head 0.15³ behind the eyes, slim torso retracted,
|
||
wrist-anchored 2-bone arms. **Soft IK applied**: shoulder→hand distance clamped to
|
||
reach×0.98 before the solve (the straight-arm singularity was the overextension shake; if it
|
||
persists, lower SoftIKFactor toward 0.95). Elbow pole: torso frame, shoulder→hand axis
|
||
PROJECTED OUT, hand weight 0.25, exp-smoothed 10/s (twitchy wrists fixed). **Torso yaw
|
||
spring**: HMD yaw base blended toward hands-derived yaw (weight 0.6 × forwardness over
|
||
0.3 m), exp-smoothed 12/s, decoupled from the head — the "chest = headset + hands" ask.
|
||
Reach budget hysteresis unchanged. Full sourced findings in `research/vr-arm-ik.md` and
|
||
`research/vr-arm-ik-deep.md` (HL:Alyx went hands-only on purpose; SkeletonIK3D deprecated →
|
||
custom SkeletonModifier3D when a real skeleton arrives; LookAtModifier3D for head→neck).
|
||
Physical torso/head capsules later. EMBODYING (view-from-character-perspective) comes after
|
||
the head/neck look override exists. **HANDOFF.md at project root is the takeover document**
|
||
(constant table verified against code — code wins on drift); keep it updated when systems
|
||
change.
|