[1167] Networked IK has massive degradation issues for some people
complete
Fizzytaco
Networked IK seems to have degradation issues for some users and just breaks after a while for them. I noticed this yesterday at the Community Meetup on the Open Beta Build 1167 with my friend MY_MAN.
His Networked IK would start off incredibly slow/delayed until it got worse and worse as the instance went on. It would take minutes at a time for him to move. His IK was completely working locally, but just wasn’t syncing for remote clients.
We could never figure out a root cause. Everything else related to players (Parameters, Voice, Objects, Udon stuff) all synced completely fine. He has a Core I9 9900K and the issue seemed unrelated to cpu load. He also has gigabit internet, and he was able to stream perfectly fine. (Ending stream also didn’t seem to affect it at all)
He also tried restarting his game and pc and re-installing the open beta. Neither improved the issue. The Networked IK issue happened in both the 80-person meetup instances and an instance with just 2 of us. The only fix was to switch back to Live. Live had no issues and synced IK perfectly for him. It’s a bug caused by the open beta seemingly.
I also recorded footage documenting the bug and our testing with it. Here’s a clip compilation of it I edited together: https://youtu.be/YO2_uCqF0l8
It should hopefully show off the issue well. The first two clips have a comparison between my remote view and his local view from his stream
(Also, apologies for the poor framerate. This beta just had really terrible performance for me)
My friend MY_MAN and myself will continue to test this with upcoming betas. Let us know if we need to try anything or if a cause is found. We’re available most days to test it. Hopefully it can get fixed!
Log In
Phasedragon
complete
This was fixed in build 2022.2.1
PinballsWizard
this now seems to be fixed for me in the latest live builds avatars with more than one stations, and in a sdk 3.0 world, and with index controllers now have ik synced properly
i don't know if the issue would reoccur with an avatar with nearly filled out parameters, mine currently are below the previous 128 limit
Tupper - VRChat Head of Community
tracked
Still occurs!
Tupper - VRChat Head of Community
complete
I believe this issue was solved in a previous update. Correct me if I'm wrong!
Fizzytaco
Tupper - VRChat Head of Community: We checked like a few AD betas ago and it wasn't fixed. And none of the patch notes have mentioned fixing it at all.
But we'll try to test it tonight and get back to you
J
Jdbye
Tupper - VRChat Head of Community: Wasn't fixed on Sunday, I can confirm that.
Fizzytaco
Tupper - VRChat Head of Community: We checked with the 1188 build and it still isn't fixed. Having 2 or more chairs/stations and index controllers still causes networked IK to stop sending. This bug is not at all fixed.
You can check for yourself with the same avatar id from before: avtr_bb8d09b5-fd7f-490e-822e-c28d014b81d9
Tupper - VRChat Head of Community
Fizzytaco: Hokay, thank you! That's the repro I was looking for. Opening back up.
Tupper - VRChat Head of Community
Merged in a post:
SDK3 worlds are broken - My avatar lags/is frozen in place for other people
J
Jdbye
Not sure what is causing this but it seems related to the new update as it only just started happening. A friend helped me test and confirm the exact behavior.
Earlier today, everything seemed fine, but I only went to one world and it was likely a SDK2 world, where the issue doesn't occur.
When I tried to get back on VRChat later in the day, everyone sees me frozen in place, they can hear me fine, my ping is fine, after I move around for about 45 seconds sometimes they can see me moving but it appears very laggy/glitchy. Most of the time people just see me frozen in place.
But in SDK2 worlds there's no issue, I can move around with no issues, everything appears fine to other players. Yet, the same issue happens in any SDK3 world I go to.
On my end, everything appears normal, there's no indication of anything wrong.
I've asked my friends and nobody else seems to suffer from the same issue that I'm having. But everyone sees me as lagging/frozen in the exact same way. I've had to give up on playing VRChat for today because it's just not playable at all in this state. And I was really looking forward to the drinking night, but there is no point in going since I won't be able to communicate with anyone.
Edit: For the record, I have tried restarting the game multiple times, I've tried reinstalling the game, I've tried doing sfc /scannow, I've tried rebooting SteamVR and my PC. There's no reason this should be happening.
Edit2: The same thing happens whether the world is US or EU so I really doubt it's my connection. I've had no issues with my connection before, and I can access everything else fine.
Edit3: I tried the beta (ik-beta) as well and it made no difference.
Edit4: Reverted to build 1160 to test and there's no issue, so it's 100% confirmed an issue with the new update.
Fizzytaco
Does your avatar have stations/avatar chairs? If so, then it's probably this issue that a friend and I discovered late into the OSC beta: https://feedback.vrchat.com/open-beta/p/1167-networked-ik-has-massive-degradation-issues-for-some-people
Basically, if your avatar has multiple stations, you're using index controllers, and you're in a 3.0 world, networked IK will fail to sync until you rejoin as a different avatar. The fix for now is to only have 1 station on an avatar until it's fixed
J
Jdbye
A friend suggested I revert to build 1160 to test and I can confirm that there is no issue in that version so it's 100% confirmed an issue with the new update.
J
Jdbye
It's the next day, and the issue is still happening. It seems like the new update broke something badly. The game is not playable like this, i have no choice but to take a break from VRChat until this is fixed.
Tupper - VRChat Head of Community
in progress
Caused by using Index controllers while you're in an avatar with greater than 1 avatar station on it.
We're probably not going to fix this for this release. We will fix it in an upcoming "2022.1.1p1" patch that'll also include some other things
Tupper - VRChat Head of Community
Unimportant sidenote: Why's it always gotta be stationssssss aaaaaaughhhhhhh
MY_MAN
Tupper - VRChat Head of Community: As I was telling Phasedragon Tupper (since we can't tag multiple people on here) thank you for your assistance on having the team look into this issue this was truly a bizarre bug and couldn't have done it without you and the teams help!
Tupper - VRChat Head of Community
We're having difficulty reproducing this issue. Can you provide more detail?
Phasedragon
Fizzytaco: This looks like a classic case of the outbound network being clogged, however there are usually protections in place to prevent udon from ever using the network so heavily that it would block IK from going out. Perhaps those protections are malfunctioning or something else is hogging the network.
To help confirm or deny this hunch, it would be useful to see what the debug menu's "Suffering" stat says. You can access the debug menu by adding --enable-debug-gui to your launch parameters, then ingame that will allow you to press rshift + tilde + 4. If you can get this issue to happen reliably, it would help if we can see what that menu shows from MY_MAN's perspective. A suffering value of 0 is normal. With the extreme desync you're experiencing, if suffering is the issue, I would expect it to be up in the tens of thousands or more.
It would also help to have logs from MY_MAN, perhaps there is something in there that would be obvious.
MY_MAN
Phasedragon: Thanks for the info Phasedragon. I am going to try out the new build as well to see if that might have fixed the issue. If not I will be sure to screenshot the debug log here for you. I'm not going to be home for another 2 hours so apologies for the delay but will get on that as soon as I am.
Fizzytaco
Tupper - VRChat Head of Community: (Also Phasedragon since I can't seem to reply to multiple people)
(I also managed to accidently delete my other comment about more details. So sorry about that)
We did some more testing tonight to get the things Phasedragon suggested and we figured somewhat of a way to reproduce the bug!
It seems it caused by switching into/loading in as MY_MAN's avatar. Here's the id for a public version of it to test: avtr_bb8d09b5-fd7f-490e-822e-c28d014b81d9
To reproduce, you need to be on the beta, in vr, and in a 3.0 world. (Desktop and 2.0 worlds don't cause the issue.) As soon as you switch into the above avatar, it should break your network IK. Both for others currently in the instance and late-joiners. Your network IK will continue to be broken for others even if you switch into a different avatar. The only way to fix it is to re-join the world as a different avatar.
We didn't catch the avatar being the issue, since we only had him switch avatars in world. We didn't have him rejoin as a different avatar.
We weren't able to figure out what on his avatar was causing it, but we atleast have a way to reproduce the issue.
I'm also the one who creates/updates his avatar and sends him the unity package to upload. So if you need any information about the avatar or want to test anything, just let me know. I already tried removing the colored text in the menus and re-uploading it to my account, but it didn't fix anything. I also use his Base and Action controllers on other avatars and those avatars are completely fine. His only unique layers are his FX and Gesture controllers.
Besides that, we did try the things Phasedragon suggested. I've attached a screenshot of MY_MAN's debug menu. The suffering value was 0 while in the affected avatar. (For the other values, Im the top person that's not in vr, he's the bottom one)
We also have a pastebin of MY_MAN's log. It's cut down to start when he leaves his home world and join's my home world: https://pastebin.com/4sQFsck9
Phasedragon
Fizzytaco: Thanks for the avatar ID! We've identified that the issue is due to having too many stations combined with index controllers. We're tracking this internally and will fix it in a future release, but for now what you can do is remove a couple of your stations and everything should be fine.
MY_MAN
Phasedragon: appreciate your assistance in finding the issue Phasedragon as we were truly stumped on what was going on and why it was affecting me in particular. But glad me and Fizzy could also be of assistance in finding this bizarre bug.
Load More
→