Ever since a few updates ago, I've been getting major FPS drops that seem to increase every time someone join. Then, I noticed that when I tab out of the game, the FPS becomes normal. And looking at task manager, VRChat is using more GPU when the game is tabbed out. Basically the reverse of what you would expect...
I'm not exactly sure what cause it specifically, but I noticed that interacting with a video player will cause that bug to happen. And joining an active instance will also cause that weird fps drop. Hiding everyone's avatars doesn't fix it, and the FPS drop is still there when I'm the only one left in the instance.
I got:
Ryzen 9 3900X
RTX 3070
64 GB RAM
Windows 10
All my drivers are up-to-date, and this also happen on fresh restart.
Video example: