Per TCL's request, I uploaded a version of my drawing retreat world with a custom material set for the terrain: https://vrchat.com/home/world/wrld_c86e0a98-0fbd-4b5a-ae01-797aa9e8e0e3 .

After uploading, the rocks outside appear dark. These are vertex lit detail meshes, if it helps. Second image is live.