It feels very punishing that I cannot change out aqua affinity (that I accidentally put in my drill in the first place) for another upgrade I found later. For how rare some of the upgrades are, it should not be permanent IMO. Would be nice if there was some way to remove or change upgrades. If it has to cost something for some reason that's fine, I just think it should be an option.

I came here to make this same request. I assumed right-clicking a new upgrade would remove the oldest upgrade first - and instead it did nothing. So now I need to make another drill, and worse, I've wasted a Carmot Stone T_T

Aa I noted in Discord a few weeks ago: I do want to change this. The question ultimately falls on how to actually do it.
The old solution was using the Smithing Table. This worked since the UI was different (a + b = c). It was natural and expected for most players that the upgrades were non-reversible.
Alternatively, since the Drill currently uses Bundle mechanics, I could also let you just take the upgrade out. It does kinda go against what I had in mind with creating more permanent upgrades for it, instead of full modularity (like the Carmot Staff).
This choice does matter, as I plan on rewriting it internally to be more data-driven/modder friendly.