New video-player backend
ArtificialVR
AVPro has many issues, ones I've noticed but certainly not an exhaustive list are:
- Relies on platform-native APIs, meaning that the only reliable format in my testing, without having every user install codecs onto their device, is H.264 with AAC audio in a .mp4 or .ts container.
- Newer codecs like AV1 or VP9 are higher efficiency, which would save bandwidth for users and hosters, and platforms like YouTube are increasingly serving H264 less and less.
- Newer codecs would also allow content-focused worlds or live events to serve higher resolution/bitrate without sacrificing file-size/bandwidth
- Inconsistencies with 5.1 or 7.1 audio.
- VRChat's implementation lacks support for multi-track audio, video, or subtitles
- Multi-language audio within one file instead of multiple files would be nice.
- Relying on individual worlds to implement their own subtitle parsing and display isn't ideal. Subtitles are crucial for those who are hard of hearing, to watch content in other languages, or to watch content with quiet dialogue.
- (However, this could also be patched by exposing the AVPro audioTracks & subtitles in Udon)
- Lacks support for LL-HLS, or WebRTC streams
- Newer livestream formats than the commonly used RTSP are more resilient to network interruption and would allow reliable playback with lower latency from more sources, like Twitch.
- WebRTC isn't as important but would allow for real-time interaction similar to how Discord screenshares allow for it.
Unity Video is also not a suitable replacement, due to
- Cross-platform support of only H264 and VP8, in mp4/webm.
- No HLS/DASH support
- No livestream support
A new video-player backend could have large improvements for the VRChat playerbase.
Some options could be
- LibVLC (https://github.com/videolan/vlc-unity) (LGPLv2.1) Supports all platforms VRChat supports, has a unity plugin.
- MPV (https://github.com/mpv-player/mpv) (GPLv2) May not support mobile platforms and requires more work to implement.
- FFMPEG, would require significant work to implement within Unity.
- Something else (?)
Obviously these options would likely need to have patented codecs like HEVC removed to avoid legal issues, and would be a noticeable amount of dev-time, but I think it would be a hugely positive change.
Related:
- https://feedback.vrchat.com/feature-requests/p/implement-libvlc-or-similar-for-vrchat-video-players-current-solutions-are-nothi
- https://feedback.vrchat.com/feature-requests/p/add-support-for-hls-dash-for-low-latency-hls-streaming
- https://feedback.vrchat.com/bug-reports/p/yt-dlp-eats-multivariant-hls-playlists-proposed-solution-included
- https://feedback.vrchat.com/sdk-bug-reports/p/document-supported-codecs-for-unity-and-avpro-video-players-on-pc-quest
- https://feedback.vrchat.com/udon/p/any-other-codec-then-h264-doesnt-work-properly
- https://feedback.vrchat.com/feature-requests/p/hls-m3u8-support-for-quest-users
- https://feedback.vrchat.com/feature-requests/p/subtitle-support-for-the-video-player-component
- https://feedback.vrchat.com/feature-requests/p/71-multichannel-audio-support
- https://feedback.vrchat.com/bug-reports/p/hls-video-stream-playback-with-avpro-player-on-pc-has-high-latency
- https://feedback.vrchat.com/udon/p/expose-multi-audio-track-selection-for-vrcavprovideoplayer-to-udon
Log In
4Bakers
I've seen worlds that majorly used to majorly rely on Video Players for their graphics butcher themselves with poorer quality baked in fallbacks because over the past few years Video Players have become abysmally unreliable.
I had aspirations to use Video Players as atlases and other cool advanced things but I had to scrap all those ambitions because they STINK!
Unity sucks. It can't hardly do anything and it cuts out terribly on weak performance.
AVPro is better but you can't use filters.
They're both unreliable as heck. I hate them.
And videoplayer frontends? They're the source of most of my VRChat frustration, most of the time. They're so overcomplicated and hard to work with in the editor. Hell. I hate it. Miserable. I kick myself everytime I have to add one to a project. God forbid I have local background music, or anything slightly complex. They shatter like glass.
「ʀᴀɪɴʏ」
Would definitely love to see some improvements to streaming. Long HLS sessions are woefully unstable and have a/v drift in VRC in my experience and is just generally challenging to make stable.
Few thoughts:
AV1/VP9 really needs decode support in HW. Obviously if the codec isn't supported in HW for decode, this could be computationally expensive for many user devices to do on CPU. Quest2/Quest Pro supports VP9, but not AV1. Unfortunately a lot of the industry kind of moved on from / skipped over VP9 hardware encoding. For example, even an RTX5000 generation NVENC does not have VP9, even a RTX3000 series NVENC does not support AV1. Real-time VP9/AV1 on CPU is very expensive. I'm personally not familiar with AMD VCE on a dGPU, but on CPU AV1 performance in general (on Raphael gen at least) is not good. Things get even weirder when you start looking at iOS devices (although, I don't think external video works anyway on them..?).
WebRTC will probably be superseded by MoQ (Media over QUIC). WebRTC video and CDNs do not generally play well together like MoQ can. WebRTC signaling layer is also a disaster and often varies from platform to platform or player to player implementation. WHIP/WHEP can address some issues, but I'm not sure it's worth it at the end of the day.
I'd personally love to have all of these, but there is a risk of excluding users on the platform in situations where these newer technologies might be the only option, and a user has no available fallback.
Absolutely would love to see LL-HLS or chunked CMAF based streaming, SRT, RTSP(T), or something come to VRC players. It could definitely be a big step forward while still utilizing codecs that are widely supported, like H264. Likely wouldn't see CDN support for SRT/RTSP, though. WebVTT could be really cool, too.
I work a lot with streaming video on the CDN side of things, so I see a lot of this first hand. Improvements are coming in this space in general, but is just slow moving, and even slower more recently with hardware costs for new gear constantly increasing to end users.
Docteh
「ʀᴀɪɴʏ」 i think they're asking for vp9 mainly as its a codec seen on youtube
Personally I'd like to see something supported where a video can be encoded in advance in a few different codecs, like do h264 and h265(HEVC) and the client will get whichever it supports. I don't know if this would be an already existing tech, like HLS or DASH, or something new.
Right now HEVC AV1 and VP9 support can be added to Windows Media Foundation via the windows store, HEVC is a paid app (1.99 USD?), and AV1/VP9 are free.