When you retrieve items from totem of holding it will not return the backpack to the curios slot. It will either put the backpack on your chest slot, or put it in your inventory and drop all the items.