It seems that this wonderful mod (which to say is absolutely essential when playing RO+RSS) have some threshold for balancing masses. That may be not an issue in stock game or with big ships, but when you try to balance the tiniest and lightest asymmetrical probe in Realism Overhaul pack... and that means NO reaction wheels - just pure RCS for control... RCSBuildAid shows that probe is balanced (zero CoM to CoT deviation), while the test flight shows that it's very close to, but actually not.
I believe the problem is with too high threshold of showing disbalance for such a tiny targets.

Can we get an option to switch to lower threshold when needed? Or an alternative digital form of showing it like dX-dZ without any threshold?

Is probably that most numbers in the GUI have a precision of two decimals (although there's a threshold for displaying arrows, if they have a magnitude lower than 0.05 they aren't displayed), I will see about adding another decimal but I don't know if I will be adding options for tweak the plugin like that, I want to keep things simple, both for me and for the user.

I can see that it can be an issue for hardcore mods that don't use reaction wheels or SAS, but how important is this imbalance though? one reason of why I didn't put many decimals in the values is that perfect balance doesn't work in practice, because your CoM shifts when you burn fuel (more so if you r vessels is asymmetrical) and then that 0.0000001 torque you had in the VAB becomes 0.1 in flight.

CoM shifts VERTICALLY (VAB craft) and that's normal. The main task is to balance things HORISONTALLY, so that the engine thrust won't produce any rotation when accelerating straight ahead.

Version 0.5 adds another decimal to torque value.