Swords will display on the side even if theres not one in the inventory
tr7zw commented
This feature just shows the last known sword a player held since it can't see the inventory of other players. Yea I could check for yourself, but then it would just desync even more. Not a bug, intended side effect.