# FLUID VR HAND ADAPTATION + Hand Tracking - HYBRID PRESENCE ENGINE (lHgpibsjgek, 1202s)

[Music] it's me n so welcome back to another VR video today we are back in Project TX well at least the makings of project TX kind of be behind the scenes stuff um we are for the first time in a video properly in a proper video I'm looking through a Quest 2 connected on link even though this build will be compatible with PC and Quest when it when it's built um that's for a reason and you will have figured that out pretty quickly uh as well before I actually use it but right now today's video we are looking at something a little bit more advanced than my usual stuff it's a new system uh it's a new system that repl place is an old one um and what I've been doing recently is I've been rearranging a lot of my cord um a lot of my old cord um kind of the the entire hybrid presence engine I'm calling it hybrid presence engine now uh because system doesn't really cover what it does it does everything from tracking to the body physics to the body pausing the elbow ik it's not really ik but it's ik it's inverse kinematic you know at a logical level uh it does all of that thing that stuff the the whole gameplay mechanics and everything and it's all modular and uh that's the new sort of layout is is completely modular it works really really well what we're looking at today though is what you've been looking at this whole time as well which is the new hand pausing system uh this is something I'm calling the fluid hand pause system uh you will have uh my words have just completely died there uh you will have played a game called Born lab uh recently in the past year or you may have at least seen it if you're a VR Enthusiast and uh in that game one of the new systems they' kind of built was the idea of avatars that you can retarget your natural body to match the Avatar's uh sort of structure to try and keep the avatars with their own natural structure rather than retargeting them to make them shaped like Q so for example here with my regular physics hands I could switch to longer arms and you can see that they kind of match my same uh my my natural arm when when my arm is at full extension my my arm is at full extension in the game even if it's longer so here we go regular hand and now for the longer one and it's it's matching that right so same thing as you would see in that game and it's full physics right so there we go but I thought I wanted to take that idea and I wanted to do it for something that you wouldn't really expect to do it for hands hand pausing so the finger posing that you're seeing here let's go back to a regular hand so it matches mine the finger posing that you're seeing here is not built in the way that normal finger pausing systems are built where you have data specifically oh no let's give me a sec sorry about that uh what happens when you tap the controls I'll show you that in a minute um when you have each finger in a regular in regular hand pause system you have each finger as actual data points inside the system and uh they kind of pulled in you kind of set your finger to a certain pae and you apply that to your character model in this system I'm not really doing that it's a little bit more complex in in terms of data I call this the fluid hand pause system and the reason being that I'm not storing specific finger pause data rather it's just sort of a blend right across the board and it doesn't actually care how many fingers you have which is very very unusual in in fact it doesn't even care how many segments you have either so here you've got 1 2 three you've got three segments on each finger um but it doesn't actually care about that that stuff is not part of the system it doesn't have dedicated finger data for each of the fingers right which means you can take these regular poses so here's my my regular poses there's only four pauses in my entire system right which is uh open hand uh grip trigger index trigger and then both triggers right that's that's all of the pauses right and of course then there's the touch sensor data that comes in and and adapts to that and adapt is the very uh is the important it's the word of the day today right uh a word of the night evening it's past 12 it's morning but there we go so what this means is because it doesn't care how many fingers I have on the hand it just it just has the same poses I can switch to different hands so here we go so now I've got three fingers and a thumb on each hand and these pauses are the same four paes that I was using just now it's the same data literally the same data it's all just coming in and applying to the character model now of course you're probably thinking okay well that's very simple you've got three fingers you can just cut one out and and and only use the data from one of them uh uh only only use the data from one of them only use the data from three of them instead of all four that's not what's actually happening you can see here each finger has a different pae right which makes sense because you only have three you already have four fingers worth of data but if I were to go to a higher Fidelity hand compared to a regular hand now I've got five fingers plus a thumb and each one is paused in independently right which is absolutely crazy the point being that it completely adapts your regular hand pause data into the uh into whatever hand you want right which is absolutely insane it's it's it's so nice it's so nice when you you've actually got it running and uh these really really like it it gives you a feeling of actually using a different hat it'll be even better in a moment in fact I've gone and extended this Beyond just the fingers you can actually have an extra thumb and again the data is separate so here we go I've got two thumbs and as you can see they do not move at exactly the same point right so here we go we've got this data here you can see that although my controls I actually control them directly like that and you can see how they're kind of completely separate they're separate things you see that it it's so cool it's so awkward you can have any number of fingers you can have an infinite number of fingers if you want and uh and and multiple thumbs and I will adapt the system as well to to control how many thumbs are actually how the thumbs control right it's going to be even cooler and that it it gets really cool because what that means is now you can have your avatars that are adaptable right where you kind of adapt your avatar to to the physical body structures and you can even adapt the handpost system to much now I've got a little bit more detail coming from the handp system and that is the first thing is really really cool is that actually tracking properly right it doesn't track perfectly because the physics rate is not match I'll just make sure that the physics rate is matching all already um uh it's it's probably the the recording that's causing a little bit of uh difference but what is added to this i' I've actually gone a little step further than what you're looking at already I've been trying to make the hand paw system a little bit more flexible a little bit more natural right so you'll notice obviously that the finger pauses themselves they're really quite nice right they work really nicely and they kind of do what I want them to do which is open up completely or just naturally claw to to to the amount that you want right but there's more going on it's actually if if you have a look at this finger for example you'll see that it's kind of tilted downwards you can see in my video that my my finger is not touching the sensor here's what happens if I touch the sensor so my finger is not touching the sensor and it's kind of pointed downwards but if I was to stretch the wrist here whoop you can see that it's actually stretching the finger out as well and that's what's going on which means that when you're naturally kind of moving your hand around you get this natural look over here right with your hands curling quite a bit and if you push them out you can see that they actually stretch your hands out right there's a little bit of natural feel feeling to that it really it's really nice it feels great to actually see that now there's even more going on than that I wanted the hands to never have a weird static pae look to them even some of the highest quality games like for example half lifee Alex they still have very static pauses when you have a pause right when you have a hand pause it just kind of stays a little bit static otherwise it feels like it's trying to do some weird animation right which uh i' I've seen it plenty of times in plenty of games right but I decided I'd add something that you would call Micro stutter right where the fingers actually do that little bit of mechanical movement that you see in your real hands when they are when they are held so I'm not actually doing anything with the triggers here and if you look very carefully you will see that that that finger did it just then there's a little bit of micro stutter happening on the fingers every every now and then there you go you can see it just there right they are very very slightly twitching and moving every now and then it's just a mechanical twitch that happens if you were to Hold Your Hand straight and you would every now and then you just see that your mortors as soon as you activate even the minute amount of force in them you'll see a little it'll suddenly switch and you can see the mortars of your fingers just kind of activating and this is something that I've I've added to this and then just try and show you as closely as possible you can kind of see it on both hands here there you can see it on each hand just having this little Twitchy twitchiness it's very very slight when you turn it sideways obviously it's a little bit more obvious there we go you can see it and it's happening on the thumb as well it's very very slight and you're not supposed to notice it when you're playing but it just gives a tiny little detail to the hands something that you just didn't expect but this system gets even more awesome ignore all the little Twitchy Twitchy stuff ignore all the little hand pausing logic of coming from controllers this stuff this stuff is really really cool when you throw the controllers away throw them and it fell right on the floor right I thought I'd hit the bed there uh one of them hit the bed but there we go right I'm going to have to wait for them to disable now oh I messed up the video but there we go right so they will they will switch off unless of course one of them's triggering hey where are you come here come here let's right I'm going to redo that bit yeah I'm going to read out imagine that never happened right so this is mess I messed up right okay we redo that b right imag you look into my eyes look into my eyes it never happened right you forgot okay okay this system gets even more cooler when you throw your controllers away throw them away boom and now I see it worked better now I've Got Hand tracking oo now why is that cool it's really flipping cool because the hand tracking comes in right the hand tracking comes in I change I I convert the hand tracking data that comes in the rotations and everything into my new system which means now I can control each finger using hand tracking and it then gets adapted into the new hand pause system it gets adapted into whatever hand I want oh things get really cool but of course I have to reach over there first so I'm going to switch over to my other Locomotion system and we'll walk over to here and look at this right I'll switch off the loo for a second go close to the oh looks like I switched it on instead I'll go close to the camera and you can see here with just the one finger I can control each of these naturally right it's just completely natural and this is the pinky finger so I'm not just cutting out that Center finger right if I were to go and extend this with the High Fidelity hand and it's really cool because the physics is also affected by this right so there we go you can have actually you can actually it's really nice but uh now you can see that each finger pauses separately to all the others right it's really really cool this is incredible cuz it feels like this is my actual hand it does actually feel now like I'm controlling five fingers and a thumb and you can see here when I point it still points with that one finger right obviously every now and then it kind of flips it out out because it's tracking a little bit separate but there we go it's one finger right it just looks right look at this it just looks incredible and of course it gets even more awkward when you've got a thumb right an extra thumb yeah the hand tracking is pretty good on this thing but there we go now I've got an extra thumb and it just works it just work it's so flipping look at this thing right I've got complete control over each finger and I've got this extra thumb right it's really re it just it feels like it's my hand it feels like it's my hand oh it's incredible right let's start walking over here uh and dude activate activate what's with my menu button today right there we go and now it gets even cooler because now even with my extra fingers I can adapt not just to the extra hand stra uh structure but to the long long arm structure as well and this is with hand post uh hand tracking right this is with hand tracking I'm able to walk around right naturally in my Game World space right it's all completely natural there we go right complete walking around and of course these are physics hands as well which means if I step right off step right up there we go I have full hand tracking controlling now and I can climb around using my full physics right it's absolutely incredible I'm going to switch away sorry I'm going to have to walk away from you for a moment just to get over here once more I'm going to keep my long arms and I will walk over here we will have something a little bit less uh awkward than the double thumb and we'll walk over here and now we'll walk over here sorry I'm just going to run for a moment and then I'll talk to you directly uh but there we go you've got this actual hand now uh that is used using hand tracking and of course we were not using hand tracking before we switched to it live by throwing the controllers twice unfortunately twice because I had to uh go and mess up first uh but there we go now if I just try not to burn my fingers on the light bulb I can actually climb right up on here and there we go full physics full hands all just works it's absolutely beautiful it is incred incredible I could switch off the Loco system get my hand over here and find the other controller and uh there we go we are now back into the regular system ready to play games really really nice it's flipping beautiful it is beautiful that is this video I hope you have enjoyed my absolutely insane ramblings it is actually a little bit cooler actually uh let me just throw the controllers once more right and uh yeah we can actually we can also see the little physics colliders on the on the thing going on as well so there we go we can Little Hook bit over here and then you can turn your hand over and there we go it's absolutely beautiful right so normal hand obviously cannot hook right uh but the little hooky hand there we go right and then you climb over and booya it is absolutely beautiful right there we go so now we'll just pick up the controllers one more time right and there we go just switch to this system now none of this uses the uh The Meta systems right this is all open XR this is all using the generic open XR system which means if your headset supports open XR and can do hand tracking it it can work with this if it doesn't support hand tracking it doesn't need to your controllers excuse me your controllers will work just fine and all of this is just adaptive it all just works just works and I mean actually just works this is why I don't do cuts in my videos to really show you that things just work with the exception of me throwing my controllers having them land on the floor and then trying once more but it just works otherwise right it is absolutely beautiful I am really enjoying working in VR once more but there we go on a quick note I want to mention financial situation is always a problem so if you're willing to support me if you'd like to support me please have a look at my patreon I am going to be increasing the amount of stuff that I put there and by increasing I mean more than zero I am going to be putting some more devlog stuff there uh you know actual rewards um and I'll be trying to get I'm I'm going to be trying to get an actual demo here look this thing is now super stable uh minus the falling of the controller everything is super stable now the hands the the physics the arms everything super duper stable it can switch between the different systems that I wanted to be able to switch from and to The Locomotion can just switch everything's properly integrated is absolutely beautiful thanks very much for watching I hope you have enjoyed and I hope you are willing to wait another week or two for the next video there is going to be so much more it's going to be absolutely amazing thanks very much for watching I will see you in the next one [Music] goodbye
