Tupper - VRChat Head of Community: Sure i can provide them but this is independent of Avatars being used. I used these two avatars simply because i just had their Unity Project open so i could quickly do some tests with them.

This happens to any and all animations in the base/additive layer, regardless of whether i use the default provided animations or custom ones. I tried using custom ones once in hopes that this might just be an issue with the stock animations (which are replaced runtime), it isn't. I can see the issue on everyone not on VR and everyone else can see these issues on everyone not on VR as well. Some users said this has been a thing since Networked IK has been added (which is from before the time i joined so i don't know if this was not a case back then)

The rex used for the movement tests:

avtr_66bf57f8-ad95-4987-8487-b131ecab642f

The kobold used for the dance test (because it has dance animations set up already)

avtr_85fd4593-385d-4e99-a59e-72dc20f31252

Tupper - VRChat Head of Community: Now that i finally had VR to test i was also able to record this with a VR and Desktop side by side.

In this side by side test i let both Desktop and VR strafe sideways to show the difference between animation qualities, you can easily and clearly see that Desktop's avatar has much reduced animation quality. It looks like the entire animation consists of just a very few keyframes (roughly 4-5 in total is my guess), you can clearly see how the up/down movement is literally just 2 keyframes between which is linearly interpolated, also all limb movement looks "blocky" and stiff. Note that the VR player here does not have a perfect animation either, it is missing the up/down hip movement (which is most likely due to seated play being used since i put down the VR headset to stabilize it... or the other thing i noticed, that Full Body VR has even better quality, this VR player has only a headset and controllers)

In this test i let the VR player run up and down for the Desktop player to watch, you can see how smooth and well animated all bones are (once again except the missing hip up/down movement due to seated play, apart from that and the stiff head which is due to the head obviously being tracked the animation looks basically almost exactly like it should)

And in this final test i let the Desktop player run for the VR player to watch, it is immediately apparent that the animations look really bad and downgraded compared to the VR player. Almost like roughly 90% of the keyframes were removed.

I still suspect this being a refresh rate issue, since VR needs a higher networked refresh rate (to keep all movements smooth) their base animations (which again should not be networked as they are just being played back and should do so for all players equally) whereas the desktop does not need a high refresh rate thus getting his animations shafted. The "easiest" way of fixing this would be simply giving everyone the same refresh rate (the VR refresh rate) but this would still be suboptimal as we'd be sending more network data for no good reason. Instead the correct fix would be utilizing the animation controllers and animations coming with the avatar and playing these back while adding the networked IK for head, hands and (if available) hips/legs ontop. This would drastically reduce network data needing to be send and received for everyone in an instance, all animation data is supplied directly by the avatar and only necessary body parts that are actually being tracked should be networked.

Desmoulins: So to be clear, I follow the example and kinda see what you mean-- but in these cases IK shouldn't be playing. Both should have their animations playing as their base (locomotion) layer defines. (with small exceptions) so there isn't really a way that keyframes would be compressed or lost. 🤔 During locomotion (using the default layers), leg IK is ignored. Head isn't, I believe it's added on. Same for hands. But you still play the animation underneath.

Of course, this is a pretty complex thing. I could be wrong. Probably requires a pretty deep dive to investigate and, well... to be blunt, the effect is so small that I had to rewatch several times to see any difference.

Definitely noted but also probably notable that this is something that likely wouldn't be super high prio.

Tupper - VRChat Head of Community: The base locomotion "should" play beneath, that's right. It doesn't it is obviously and visibly networked. You can see this when people get framerate lags or freeze, their animation stops dead in its tracks rather than continuing or defaulting to what your client would think is happening (standing or running on the spot).

Also note that the shown examples had still quite good quality (probably because i was very close to the watcher), the quality degrades over distance and possibly with framerate. Since both of the clients had a perfectly stable framerate you didn't see it that much. But in general in games and social places its extremely noticeable because basically everyone is just sliding around, much much worse than shown in these examples, on top of it becoming even worse at distance.

It might not sound like a high priority thing but if just the unnecessary networking part is right then you and everyone around you on desktop (as well as partial VR) is essentially sending a lot of unnecessary data for bones that should be playing the locomotion animations instead. Count this up to a full lobby and then scale it to all the players in every instance across VRChat and we are looking at a massive amount of waste data just bogging down everything for no reason... i don't think this is a low priority issue. I also don't think it would require a deep dive, this really just looks like a matter of disabling networked IK in the correct situations.

Desktop: Except head and grabbing, basically never network unless standing.

VR: Should simply be fully networked at all times just like it is right now (no change) since it works well for VR users. (unless of course you disable tracking of parts in the layer or animation)

Tupper - VRChat Head of Community: Going through all of this with some math experiment reveals quite a shocking amount of bandwidth waste.

Very technical stuff coming:

Examples

Quaternion Rotations = <float x, float y, float z, float w>

Vector3 Positions = <float x, float y, float z>

float = 4 bytes

float = 8 bytes (full)

Quaternion has 4 floats, this equals to 4 x 4 bytes or 4 x 8 bytes respectively. Total: 16 / 32 bytes.

Vector3 has 3 floats, this equals to at least 3 x 8 bytes. Total: 24 bytes.

There are at least 5 bones MINIMUM that need rotations and positions networked: Head, Hands, Feet.

5 x (16 + 24) = 200 bytes.

OR

5 x (32 + 24) = 280 bytes.

Assuming we have a decent network refresh rate of 30/s and each refresh using 200 bytes minimum and 280 bytes maximum.

30 x 200 = 6000 bytes.

OR

30 x 280 = 8400 bytes.

1024 bytes are one kilobyte this means:

6000 / 1024 = 5.86KB

OR

8400 / 1024 = 7.8125KB

This means we are sending a bit under 6KB in best case and a bit under 8KB every second in worst case just to network IK for the 5 most basic bones.

But it doesn't stop there, from what i can tell network IK doesn't just network these 5 bones but basically all major bones: Head, Chest, Hip, Shoulders, Upper Arms, Lower Arms, Hands, Upper Legs, Lower Legs, Feet.

This would total the bone count to: 17. If we repeat this experiment:

17 x (16 + 24) = 680 bytes

OR

17 x (32 + 24) = 962 bytes

This means worst case we are already almost sending 1KB per single refresh.

30 x 680 = 20400 bytes

OR

30 x 962 = 28560 bytes

With all 17 bones being networked we are looking at roughly 19.92KB best case and 27.89KB worst case per second being sent.

But this is assuming we'd only send it and we're alone. An average instance has 10 players. With 10 players on average we are looking at 58.6KB in best case (5 bones) all the way up to 278.9KB in the absolute worst case (with 17 bones) being shoved around at any given time to network everyone's IK.

VRChat doesn't just consist of 10 players but has probably around 10000 to 20000 players online at any given time. We are looking at ~58MB of IK networking data as the bare minimum, assuming we use the 5 bones example and only 10000 players. This can go all the way up to ~279MB for 17 bones and 10000 players, even double that for 20000 players.

This is an insame amount of data being shoved around either from player to player, or if a server handles delivering the IK data, server bandwidth being used. This example assumes that everyone is using VR, no Desktop players at all.

VR can save the foot IK whenever not standing which saves us 56 bytes (or 40 bytes) per refresh which saves a good chunk of data. Technically the foot IK can be done completely and individually by each player in the instance as the IK calculation is the same on every client.

Desktop users on the other hand are a special case as described their refresh rate seems a good chunk lower resulting in these bad animations, an obvious fix would be raising their refresh rate to VR levels but then we'd be looking at the above calculations for Desktop users too. Instead what we could do for Desktop users would be networking head only and hands only whenever necessary (holding an item). They don't need to network their hands or feet unless they are doing something with it. Desktop users also can only grab a single item, reducing the count to 2 bones that need networking (unless Desktop users get the ability to grab 2 items). Everything else can be a simple locomotion layer playing beneath.

Putting all of this together we can save 3360 bytes (2400 bytes) per VR user and 5040 bytes (3600 bytes) per Desktop user most of the time assuming only 5 bones are networked, with 17 bones being a lot more obviously.

So fixing this (and doing it properly) would save a lot of bandwidth for everyone and also improve animation quality to the absolute maximum for everyone as well.