Enabling legacy blendshape normals in the model import settings reduced the size of my avatar from 120+MB to less than 25MB. There should probably be an auto fix button that does this for you in the build and publish menu.