DarkSwordsman
This feature request is asking VRChat to implement a "universal" system for streaming video to other users, regardless of the world or instance.
## Concept
The idea is to expose an API endpoint that someone can stream to, kinda like Twitch, which can be projected onto an object in game. This would allow other users to see content from other players near real-time, creating another layer of social interaction. Think of it like a similar social mechanic as someone IRL using their phone to show you something.
So for example: I open my OBS and set my stream URL to
vrchat.com/api/stream
and I have a stream key that VRChat uses to validate. I then start my stream, which allows me to spawn a tablet or something that other people can use to watch. This could either be a single object that everyone sees, or some way to view things on your
own
tablet from another friend.
Some other thoughts:
## Deployment:
mediamtx is an open source stream relay server.
This is something that can be configured or wrapped with other software, and deployed for a relatively low cost, only requiring a 1 Gbps connection on each VM that can probably handle 2 to 4 full instances of people, depending on bitrate. It simply copies the stream without transcoding (similar to how discord does it), so it's relatively low overhead.
Current video players do accept RTSPT. In the context of mediamtx: This means sending an RTMP stream over port 1935, then the end-user pulls with rtspt on 8554. In my case: I have an OVH VPS on the east coast 2,000 miles away from me. I am able to generally manage < 1 second of latency, even down to 500 ms on a good day. Typically 1.5-2 on the worst days.
Of course, if that costs too much engineering time or resources, VRChat could utilize Amazon's streaming service, or even possibly partner with VRCDN to provide a video backend.
## In-Game Experience:
Probably similar to the camera page, a way to just spawn in an object (tablet) to stream to, or enable streaming on your user object.
If we went the user object route, other users can select your user pill in game or profile through the menu. And similar to how we have different options on the user profile page when a user is in your instance, you should be able to see if they have an available stream to watch.
In game, a user must start a stream manually. What does this mean?
I was thinking we can just make it available whenever the API is being sent a stream, but this could be bad for security reasons (like accidentally streaming your entire desktop without realizing it, because you were streaming earlier).
So you could have a setting where the backend may still accept the stream (for stability reasons, since restarting a stream is more manual than just turning it off in game), but the user can configure whether it's available for other users. This could be after leaving an instance, or even a timer of "are you still there?" to confirm the stream is still running.
Brownie Points: OBS
does
have a local websocket API you can use. Steamer.bot uses it. So for example, you could have OBS running, but configure it's streaming state from in-game via that API, without having to use your overlays.
## Safety
Default "opt in". The mechanics of this system would require other users to explicitly subscribe to their stream. Even to the point that you may want to
stop
people from receiving a stream if they are physically separated from other users. This can help save bandwidth as well.
This includes mechanics around audio range. We could perhaps do an ear-muffs style interface for "other streams", this way users can't disrupt social experiences, though that is mostly solved already by being "opt-in".
## Other Software that Exists:
I can't remember the name of the company, but LSMedia was testing out RTSP streaming where users in-world could interact with the same computer screen as the host. But I am not sure how far this has gotten. It still apparently works I think, but it's limited to subscribers of LSMedia IIRC.
## Day Dreaming about the Future:
A small scenario:
A user likes to teach people things, or even holds classes for a thing in VRChat. This could enable them to almost have an in-game whiteboard or display to project things on, same as how they do it in school! They can do this in any world they choose, since the system is universal. They would be able to share concepts with people real-time, giving them a much more effective visual aid, rather than relying on pens in-game.