Currently, when hand tracking is lost for a given hand, it seems to play an animation where the hand goes from straight in front of you down to your sides. This leads to a sudden janky-looking jump of the hand position.
My suggestion is take the last known position of the hand and interpolate between that position and the position where the hand is at your side. I think it should even take into account the last known posture of the fingers and interpolate each bone angle towards a relaxed posture once it finishes the motion.
I think it would further improve the appearance of the transition if the velocity of the motion was modulated using an "Ease in" curve rather than linearly interpolating. This is to minimize an immediate significant change in motion when tracking is briefly lost.
Lastly, I also think that the final posture of the hands when tracking is lost should be changed to match how the hands look on PC when using mouse and keyboard (i.e. a relaxed posture).
Please see my video for a demonstration and fuller explanation: