Wearing a witch hat prevents a Witch from becoming hostile towards the player and gives an opportunity to trade as long as the player is wearing the hat. When you're removing the hat and putting it back again the witch will still be hostile and disables the trading feature.

Perhaps whenever the player wears the hat again while being in the same room as the witch, they'll turn passive and not aggressive?