* Place a spawnpoint at -500,0,0

* Enter play mode

* Teleport to 0,0,0

* Your player will slowly drift sideways

It seems that Character Controller expects to be a root game object, unlike ClientSim which puts it inside a parent that is set to the spawn position. Moving the PlayerController object to the scene root immediately solves the issue.