This is a issue that exists for a longer time now and I've mostly experienced this issue in the Void Club. Tho I have rarely seen it in different maps too.
From my experience it randomly happens with 20+ people in one instance / lobby and it can happen within seconds after creating a new lobby if enough people join.
Example:
We had 30+ people in the Void Club and suddenly it starts lagging everyone out when someone joins in (bad framedrops or even freezing). At first I thought it was avatars causing it but I've experienced it alot and come to the conclusion that it has to be people joining the lobby. It looks like a network syncing issue or something related to music player.
First time I experienced it was after the VRC Client upgrade to Unity 2017.
I already checked the logs with Lycon but there is nothing that would show whats causing the lag.