I have no idea what causes this but apparently in instances where there is 20 or more people, the camera starts to jitter when moving with the mouse. I haven't seen anyone reported this besides 1 person (https://feedback.vrchat.com/bug-reports/p/issues-with-camera-movement). It becomes extremely noticeable when moving diagonally.
This can also happen in the current beta build of VRChat. This never happened before.