When joining a server or loading singleplayer, the first held weapon can't swing attack, and pressing one of the numbers doesn't actually change to the selected hotbar slot, instead loading the first slot's item properly. It can then be switched normally.