I saw an issue posted on the infused pickaxe mining in 3x3 after unequipped, I think that is part of a larger issue of items that apply perk-like effects not undoing the effects when unequipped in specific ways.

So far these are the methods I have seen to undo applied perk effects correctly
-switching items using scrollwheel
-items taken off from inventory only if replaced by an item dragged from cursor

Ways to remove item that do not undo perk effects
-taking the item out from inventory without an item selected on cursor to replace it
-tool breaking mid use
-item removed by /clear

If the perk effects are still active, holding the item again will not stack effects.
Shifting/irradiant stars seem to only hide from perk statistics. effects are still present.
The only way I've seen to undo the item's effects is to re-equip the item and take it off in a way that works.

as of AS version 1.12.5 on forge version 31.2.9