Body Tracking on Quest
complete
g
greg_new
My name is Greg and I am the author of Driver4VR. Software that provides a way for body tracking with other devices than Vive, like for example Kinect or NoloVR Controllers.
Body Tracking is now possible only on PC version but we could get it easily for Quest with some simple socket protocol over wi-fi. It can be either open source or dedicated to VRChat. Of course, developers would not need to bother with protocol directly but we would write some wrapper for that for c++, c#. Same as currently OpenVR API.
I am working currently with SlimeVR team to use their protocol based on websockets with other methods of tracking like Vive Trackers, Kinect, Nolo VR, Webcam, PS Move in VRWorkout app. Once demo is done we will try to formalize those and make available to everyone.
What do you think about that? Perhaps someone from VRChat team would like to be engaged in this?
Best,
Greg
Log In
Fax
marked this post as
complete
Earlier this month we released an update that adds OSC tracking support for Oculus Quest!🎉
> Using an external program, you can read tracking data from any hardware you can think of, handle that data, and then transmit it to VRChat using a set of OSC endpoints.
Thank you for your request, and thank you to everyone on the SlimeVR team!
Please let us know if you have any other feedback.
thebutlah
SlimeVR developer here. We are working on building up a device-agnostic FBT protocol, based on flatbuffers and websockets. We would happily expose an OSC API and send VRChat position and quaternion data, if that is the preferred approach. We already support owotrack, and have a lot of the initial work done for supporting vive trackers too, so we plan to be device-agnostic. We are also open source. We expose 9dof trackers just like vive currently does, but this time over an api not based on OpenVR and agnostic to the actual tracking hardware.
We already have proof that FBT can work on quest, as the developer of VRWorkout is integrating it into their application and has shown it working.
Now that the Fullbody update and PhysBones updates in VRChat have gone live, full body tracking is the only PCVR feature missing (other than higher fidelity graphics & custom shaders, of course). Would it make sense to collaborate on adding support?
T
Tsuki Haiiro
I tested FBT with just a webcam and although the functionality is still limited (doesn't detect sitting very well) it is sill a fun feature that should be implemented in OQ2 once the function is working smoothly. That being said, it's already ready for a beta test!
g
greg_new
Tsuki Haiiro: Hi. It is not the point if one method works well or not but to create open standard that allow integrate any method.
NotAKidoS
Could be integrated into the OSC thing that was shown off in the dev stream. Looking back, seems it would work well.
Would be very cool.
g
greg_new
NotAKidoS: Not sure what you mean by OCR?
NotAKidoS
greg_new: I meant OSC data, whoops.
Part of the Dev Labs section of the developer stream:
g
greg_new
NotAKidoS: Yes. That will be probably the way if they add it. Do you have any link to this OSC data?
NotAKidoS
greg_new: VRChat released an open beta with OSC support. Sadly it doesn't have everything you need available for trackers to work properly, but they've said it is planned.
You could probably animate the legs with each limbs rotation and a ton of floats, but might be best to wait for proper support.
MrDummy_NL
Aside, Driver4VR is not free, so it's kinda issue for some people. There are open source solutions already (alternatives) and we should also help those people who work on free solutions. Don't limit it only on Driver4VR. Also because VRChat is free.
Open source must fully supported, Driver4VR is not open source, it's paid software, so it will stay closed.
Thank you.
g
greg_new
MrDummy_NL: I was saying exactly about this. A open protocol that anyone can use. Then it is up to you to take whatever software you want. Either paid or not.
AuroraNemoia
As I see it, the team's focus with Quest has been more to make it an enjoyable experience than to try and bring it to feature parity with PC at all costs. There are so many things missing, dynamic bones, audio on avatars, custom shaders, we just now got the camera.
Full-body in VRChat PC is still limited to SteamVR, if it were to be added to Quest, it should also be on the Oculus PC release, and it should be as seamless and easy as it's SteamVR counterpart, if it requires messing with server IPs and a bunch of settings, most people are not going to care.
Also, before adding any sort of support for full-body on additional platforms, there are long-standing bugs that should be fixed with the de facto SteamVR solution, the local avatar pose you see for yourself will wildly stretch and compress to align to your viewpoint compared to what is sent to other players. There's no in-game instructions on how to calibrate into an avatar, and players assume their game is broken when their model is t-posing and they try to reinstall and clear everything before being told about pushing in both triggers.
I want FBT on Quest, but there are more pressing matters beforehand. Though that doesn't mean every developer won't be at the ready for the day VRChat wants to add this.
MrDummy_NL
AuroraNemoia: As you are aware Quest 2 has already limited CPU power compared to normal PC, FBT is much more work/data and so also heavier for Quest 2. We need some extra hardware help to make sure Quest 2 will not affected too much with FBT on.
Otherwise it's still big no-no.
L
Lejonkatten
MrDummy_NL: This is actually not entirely true, majority of FBT solutions require little to no processing power, not sure if you were aware
MrDummy_NL
Lejonkatten: I will wait for results, because i know already by PC, the FBT uses some % processing power so it's not really lightweight.
g
greg_new
MrDummy_NL: You are wrong here. The whole processing would be on PC or Phone. There would be no processing required to manage trackers on Quest side. But I must agree that it might take some power to manage it in avatar.
g
greg_new
AuroraNemoia: There are always bugs but it should not stop from moving forward. There is no need to add it to Oculus PC cause there is simply SteamVR already working.
「Marsy」
yush i want
g
greg_new
「Marsy」: I hope that will work anytime soon :)
「Marsy」
greg_new: yesh
∗Buns∗
If it were open source, even other apps could take advantage of this (other games, other fbt software, etc). Either way it would be neat to see.
Also if you happen to see this Greg, it might be worth writing to Oculus/Meta about implementing into the quest directly. Then games could support it with no effort.
g
greg_new
∗Buns∗: I might try that but it might take forever.
The advantage of my proposal is that it is open and super easy to implement for everyone. I am talking of course about exchanging tracker data, I am not familiar much about implementation of avatars.
p
pacapoko
As someone who uses Driver4VR on PC, I can vouch that it's worth the effort of getting it working on Quest! Even if it's just for novelty, it's a fun addition.
g
greg_new
pacapoko: Thanks. I only repeat that this api/protocol should be open and then anyone can add body tracking option.
Greg
llealloo
This sounds really cool. Keep up the great work!
g
greg_new
llealloo: Thanks. Anyway it is not up to me only. Even open and supported standard is not much of benefit if no one uses it.
Load More
→