Hide the extra quest button by shift clicking it, or middle clicking etc.
Don't unhide until the item is used from within the inventory or using the objective tracker button.

Adding more handlers to the button, which is managed by secure code, is a nuissance to get right. Tracking when the quest item has been used is also going to be a proper pain, as the spell and item have no real correlation.

With the three options I've added to adjust when the button should be shown ought to be sufficient enough to keep the button only shown when relevant.

That's more than okay, they were all just suggestions anyway.
Thank you for this addon and the time/effort you put into maintaining it.