Using masks to get transparency only works for photos right now, videos rely on green screen which can be a bit imperfect at times (green emissives interfering, etc).
LIV (the vtuber app) achieves transparency when you capture their desktop window with OBS, any chance this functionality could be added to VRChat?