This is specifically an issue with grabbing, not colliding, so far as I can tell.
Multiple people have tried using an animation to toggle "Is Animated" at runtime (i.e. disable that while grabbing, or only enable while a specific animation is happening) but it appears to make no difference at runtime.
GIF illustrating the issue courtesy of Joshuarox100.