Implement LibVLC or Similar for VRChat Video Players, Current Solutions are Nothing but Pain for Live or Fragmented Video
AmethystVR
Can you please bake in LibVLC (https://www.videolan.org/developers/unity.html) or ffmpeg or something similar so we can have functional live streams and/or adaptive bitrate video streaming🙏
If the legal team doesn't agree with it, build a library without proprietary / legally problematic codec support and hand it back over to the system decoders if it can't be handled. Having all these issues fixed even only with VP9/AV1/Opus/Flac codecs would be so so nice! YouTube hardly serves H.264 anymore and we're on a workaround just to have any playback.
Here are the issues I've found so far when working with video content in VRChat:
- No seeking support in Live streams
- Live streams or any video without fixed duration does not work in Unity player
- AVPro has significantly worse audio quality than Unity, can be easily discerned by trying "binaural audio" videos back to back, no it's not a difference in streams from yt-dlp, tested with identical files from a server I control
- HLS streams have an insane amount of latency (~45 seconds to a minute)
- RTSP streams do not work on quest (they should as of another Canny post I found? but they don't)
- RTMP streams do not work on PC, do work on Quest
- Livestreams often flicker rapidly when game lags if the worlds do not blit textures, was an update around a year ago that caused this iirc
- HLS streams are almost always missing audio unless it's MPEG-TS/H264/AAC, which is about as outdated and inefficient as it gets
- HLS streams with fMP4 segments have stuttery audio
- Adaptive bitrate HLS + DASH streams do not properly switch between codecs if computer is lacking support, it just doesn't play for some users
- Livestreams are often upside-down at random in older worlds, is world-specific not sure why
- VP9/MP4 playback does not work in AVPro, works in Unity
- No seeking support in webm files for AVPro or Unity
- Fragmented MP4 files do not play back at all (https://feedback.vrchat.com/bug-reports/p/video-players-video-chunking-not-working-for-mp4-and-webm)
- HLS playlist files have been the source of 3x untrusted URL exploits in the past that I'm aware of due to passing them directly to Windows Media Foundation to deal with
- No adaptive resolution streaming from YouTube, capped at 1080p and relying on pre-muxed file workaround that broke for days a while back
- DASH streams with slightly offset clocks between server and client do not buffer properly, it just stutters indefinitely
- Video players do not fall back to software decoding if color profile is not NV12 (https://feedback.vrchat.com/bug-reports/p/when-i-play-a-yuv444-video-in-avpro-player-and-exit-vrchat-it-freezes)
- Video players do not support subtitles at all, relying on custom implementations per-world and server infrastructure
- Video players are dependent on system codec support, often needing extension packs from MS Store
- No WebRTC support, would allow for very low latency livestreams & real-time interaction
- No option to globally adjust video player volume independently of world
There are for sure more, these are the only ones I can remember off the top of my head. VLC has none of these issues, neither do any web browsers.
Log In
syncpulse
Another one for audio pain is the difference in handling multi-channel audio between avpro and unity players. On unity you can do multiple audio tracks in the file, on avpro you cannot, but then you are limited to (iirc) 2ch per track on unity, whereas on avpro you can do 6 or 8 channels depending on codec.
This is of course in addition to the other avpro audio issues such as inability to use filters, which has been left without any solution from VRC for years now. Notable canny post: https://feedback.vrchat.com/sdk-bug-reports/p/proposal-for-fixing-audio-filters-eg-low-pass-support-for-avpro
Also anecdotally unity player is susceptible to bad stuttering when the client hitches, even small hitches in response to normal VRC events like loading avatars or opening the menu.
AmethystVR
Addressing some comments I've heard:
>> HLS Streams have an insane amount of latency (~45 seconds to a minute)
> This is because AVPro has buffering enabled for that player.
HLS by design has ~45 seconds of latency, I have found no way to reduce buffering further without LL-HLS which is not supported at all and does not work properly. This is why Twitch/YouTube have separate Low Latency modes
> RTSP streams on quest work if you use rtspt ( see hyperbeam )
Need to personally verify this - but regardless, you shouldn't have to use 2 different URLs, it should just work, especially if my server is advertising the fact that UDP is not available. Disabling UDP is not valid imo, UDP has many benefits.
> Avpro does actually support some subtitle tracks, but vrc doesn't expose them at all to userland
Regardless if AVPro exposes it or not, I shouldn't have to write custom code to use it. Subtitles should just work and be rendered over the video automatically
>> AVPro has significantly worse audio quality than Unity, can be easily discerned by trying "binaural audio" videos back to back, no it's not a difference in streams from yt-dlp
> They may be referring to how Unity audio source is one object with 2 channels that are spatialized differently from AVPro
I've tried removing the 3 channel speaker setups and replacing them with an audio source that's identical to Unity player, after messing with it for hours I've found nothing that helps. The difference can also be heard as lack of bass / lower frequencies too, I suspect AVPro is a large reason people complain about the Quest headsets not having enough bass in the speakers. Spatialization is mainly done through higher frequencies, so I suspect there's some down-sampling or filtering going on that's cutting both low and high frequencies, but I'm not a sound expert so I don't know
Three additional issues I didn't manage to remember for the list:
- Static DASH / HLS VOD manifests are treated as live video when they are not, and come with all of the problems related to that.
- There is no clear indication that a livestream is loading, the screen just stays black as soon as it's passed to Media Foundation after yt-dlp resolves the URL.
- This is kind of nothing but HTTP headers from live video through Media Foundation are much leakier than the anonymized chrome user agent that yt-dlp uses, I can identify exact windows builds & also often standalone headset types & android versions
P.S. beware, it's real :3
D
Docteh
AmethystVR The suggestion for disabling UDP on RTSP is due to how well the RTSP protocol handles NAT routing (not really). The client asks the server to connect on a certain port, and most clients these days are behind a NAT setup. So with UDP enabled, it works fine on my LAN, but whenever not on the LAN the UDP packets never actually arrive. I've heard of some routers having the ability to rewrite outgoing RTSP traffic, but I've only heard about that feature because its a feature that can crash certain routers.
Its this conflicting requirement that requires two URLs for the most part.
Windows if given opportunity to do UDP will not try TCP
Android currently requires that UDP is offered, and will not accept to be told to only do TCP.
I hacked my own copy of MediaMTX awhile back
AmethystVR
Docteh Huh, maybe my router just handles it and I'm not aware of it. I will have to have some other people test the streams that work for me, I appreciate the comment <3
AmethystVR
Docteh Out of curiosity, does rtsp://mtx.7z.wtf/amethyst work for you in VLC? It's working for me right now and the only enabled protocol in MediaMTX is UDP. Will try to leave the stream up for next couple days if anyone else wants to test.
It's also inside of Docker, the host is not in my LAN, and I'm behind T-Mobile CGNAT with additional NAT on the router.. supposed to be worst case scenario. I'm also quite confident it's actually UDP because I see the occasional packet loss that is not present with TCP and a lot of UDP traffic in WireShark. It's also working for 2x friends of mine too.
AmethystVR
Still working after hours for me w/o disconnection and several friends..
AmethystVR
Docker has a hard-coded 90 second timeout in the proxy system, this seems like it could be related to what you're seeing in regards to UDP/NAT issues, I don't think it's actually an issue once you take docker-proxy out of the equation.
I believe the problem is coming from one side timing out and not remembering you, and the other side still tracking that connection and trying to forward it to a port that the other side doesn't remember.
D
Docteh
AmethystVR Thanks for the test stream. Looks like whatever it was told to play isn't playing properly, but a stream of a broken video object in chrome is enough to work with :D
I tried VLC and Windows Media Player and VRChat, I have LAV Filters installed, so the user agent from WMP is "Lavf59.34.101" I should probably update that.
I'm noticing from packet sniffing that the first two are actually sending RTP/RTCP packets between SETUP and PLAY. Basically some UDP packets to your port 8000 and 8001. VRChat with the "WMPlayer/12.00..." user agent is not.
VRChat on a Quest 3S with Exoplayer also doesn't send these outbound packets, so with VRChat not helping the connection, the server tries to send packets to the client_port(s) specified in SETUP, and they don't go through the NAT
D
Docteh
> Livestreams are often upside-down at random in older worlds, is world-specific not sure why
Usharpvideo uses unityvideo for regular, and the live option uses avpro, my understanding from looking at the usharpvideo git history a year ago (probably a good idea to double check this), on android the video texture is flipped from normal and it is dealt with in shaders.
> RTSP streams do not work on quest (they should as of another Canny post I found? but they don't)
All of the video player prefabs I've seen give up after 10 seconds, with how exoplayer is being told to act by avpro/VRChat it takes almost 10 seconds to connect and be working. There are other details, but the typical 10 second general time out can't be worked around server side.
D
Docteh
Two bugs filed asking for udp to be disabled for rtsp.
AmethystVR
Docteh The video flip was primarily a PC bug for me, but old versions of USharpVideo do seem like the culprit here, if I could edit that point out from my post I would.
Having them disable UDP would be awesome :)