Avatars that are unavailable can't be removed from your favorite avatars list.

Also, it seems that the UI behaves differently depending on if the avatar was privated or deleted.