It feels very punishing that I cannot change out aqua affinity (that I accidentally put in my drill in the first place) for another upgrade I found later. For how rare some of the upgrades are, it should not be permanent IMO. Would be nice if there was some way to remove or change upgrades. If it has to cost something for some reason that's fine, I just think it should be an option.

I came here to make this same request. I assumed right-clicking a new upgrade would remove the oldest upgrade first - and instead it did nothing. So now I need to make another drill, and worse, I've wasted a Carmot Stone T_T

Aa I noted in Discord a few weeks ago: I do want to change this. The question ultimately falls on how to actually do it.
The old solution was using the Smithing Table. This worked since the UI was different (a + b = c). It was natural and expected for most players that the upgrades were non-reversible.
Alternatively, since the Drill currently uses Bundle mechanics, I could also let you just take the upgrade out. It does kinda go against what I had in mind with creating more permanent upgrades for it, instead of full modularity (like the Carmot Staff).
This choice does matter, as I plan on rewriting it internally to be more data-driven/modder friendly.

For now as of 0.22.0 I have made it so that upgrades have to be made with the Smithing Table. This should resolve some of the expectations that the upgrades aren't something easily swappable.

I am still looking into whether it is appropriate to also let you remove upgrades. Ultimately it will cost you the upgrade item regardless of how this is solved (E.G. replacing the oldest item, replacing a specific item, removing both, etc.).

I think a better solution ultimately is underlining this consequence (the upgrade being permanent), and providing better info to the effects so that the player can make a better choice.