This happens even if there is no portal in the current world.
DarianLStephens commented
There seemed to be a bug before where your hand, or items held by them, wouldn't render at all.
That seems to be fixed, but it looks like FOV wasn't taken in to account. This is also visible if you sprint or do anything else that changes FOV in normal gameplay.
