Tested with a Computronics Tape Drive. When I right click a paint brush on the block, it properly changes colour, but the GUI still opens. I think it should not open the GUI when the paint brush successfully recoloured the block.
Mind that I am returning false from recolourBlock to make the paint brush not take damage. It shouldn't open the GUI anyway.

The paintbrush is supposed to take damage. If you're returning false, it's your own fault - the API expects you to return true if the paintbrush is doing anything.

How are we supposed to know otherwise if the paintbrush does anything on a given block?

There are multiple mods doing it this way, see for instance OpenComputers.