While this could be multiple lucky coincidences, I found myself sitting around while nearby avatars downloaded for multiple minutes without much or any download progress. I logged off, swapped to live and joined the same instance and they downloaded at the expected speed. Swapping back to the beta again yielded the same results.

I had to walk through a crowd of 15 or so people to get to the people I wanted to talk to and a few of them did load, possibly the download priority didn't change as I moved away?