Alot of the time, with the drill upgraded with all three speed upgrades, I find it difficult doing any intricate work in any of my buildings without blasting a hole in it, even worse when I have the 3x3 or 5x5 upgrades and forgot to take them out.

This suggestion would be to have either a filter upgrade that can take an AA filter item that would either blacklist or whitelist blocks from being broken by the drill or a specific filter slot aside from the standard up grade slots.

An example, as I have built most of my buildings using Quarks marble (Modified through chisel), I would want to naturally have this on a blacklist of items that I can no longer break freely. Holding shift, that moves to 1x1 mode anyhow could also naturally ignore the filter, allowing precision breaking of blocks only when you want to.

Another use case: Working with redstone, I always find myself not just breaking the redstone but also the block underneath it. This is super annoying when that item drops as a different type (Stone -> Cobble for example). Adding stone to the filter would be so handy.

No, I don't like this, sorry.
Just take the upgrades you don't need out.

@Ellpeck No need to be sorry about it :-) I manage just fine doing exactly what you say (until times like last night when I have nearly 6000 blocks to break that are all mixed up with other blocks).