Runtime avatar performance profiling and ranking
Oli__
This is a suggestion for a performance ranking that's based on runtime measurements taken from an avatar's performance in-game on the client's system (hereafter referred to as "runtime ranking"), rather than a formula to predict avatar performance in advance ("predictive ranking").
If VRChat can measure the actual runtime performance of an avatar (CPU time, render time, RAM and VRAM usage, possibly other "bottom-line" metrics) and determine which resource is bottlenecked on the player's system, then the worst performers can be turned off based on actual performance.
- This would allow players to have as many hidden toggles as they want, provided they stay within the bottom-line metrics like download size and runtime performance. If they're not being rendered, then they're not affecting the runtime profiling, and the avatar will still rank well according to runtime ranking.
- This would eliminate the gaps in the predictive ranking system. VRAM usage and texture size is a systemic problem right now that the predictive performance system doesn't account for, but if that problem was patched, I believe that a new systemic problem would appear somewhere else where there's a gap in the predictive ranking system. Runtime performance measurement could eliminate this game of whack-a-mole.
- This suggestion is based on my day job work as a platform/infrastructure engineer for cloud/SaaS companies. My rule of thumb is to always base system alerts on metrics that are measured as close to the "output" as possible - server response time and error rates - rather than attempting to monitor for potential root-causes, such as disk space. These "root-cause" issues may not actually result in an outage, but would contribute to alert fatigue through false positives, and a collection of potential-root-cause style alerts might not actually catch an outage if they weren't watching the correct root cause (false negatives).
I realize there are many difficulties with such a system. For example, when avatars are disabled, it would no longer be possible to measure performance, making it difficult to decide whether the avatar can be turned back on. I'm also not familiar with whether Unity gives visibility into things like how long it takes to render a GameObject hierarchy, or even whether that's a valid thing to attempt to measure in a 3D scene. But I know that the VRChat engineering team has a strong history of taking on difficult problems like this from first principles, such as with avatar dynamics.
I hope this suggestion is helpful, and thank you for reading!
Log In
Arctic_SnowyFox
there could also be a separate performance rank that is crowd sourced where you compare avatar frame time to the average avatar frame time for all avatars for that user since the last time they changes a component in their PC (easily detectable in most cases). or you have a performance benchmark that a user can do that the score of that is the reference for the measured avatar frame time. this would allow the avatar frame times to be standardized into a globally applicable metric and be averaged out over all users seeing an avi. this metric is or can be converted to an additional crowd sourced performance metric based on real world data
Oli__
I was thinking about this a bit more, and wanted to write down thoughts somewhere.
I noticed this line from the Avatar Fallback System doc:
> "Show Avatar" for Very Poor avatars functionality may be removed in the future, and Very Poor avatars may be removed from Quest entirely.
It certainly seems like there could be functionality that avatars could have, which would simply not be available on Quest at all! The current ranking seems like a pretty good fit for that.
Runtime performance ranking might work better as a net new system, another reason why fallback avatars might be shown, the same way the download size limit works. If your avatar is the worst performer in the room, and the framerate is too low, then you get disabled, repeat until framerate improves to an acceptable (user-configurable?) minimum.