For example: Crossbow at power 1 or 2 its ok but beyond that its unbalanced when combined with piercing and multishot, so the crossbow should only accept power level 2.
Another example is: Sharpness lvl 5 and smite lvl 5 combination being broken, so the level of enchants should be limited like this: Sharpness lvl 5 and Smite lvl1 or Smite 5 and sharpness lvl 1 cause having both enchantments at level 5 is broken, so only having 1 a lvl 5 and the rest at lvl 1 should help on balancing things out.

This isn't very feasible to implement as the max level check for an enchantment doesn't include any context about what is already on an item.

I don't want to make invasive changes required for adding such context since it's not going to be compatible with a lot of mods which are unaware of my changes.

I can see some combinations being unbalanced, but I'd rather recommend getting / making a mod with dedicated enchantments that doesn't allow for levels that high, like a dedicated crossbow power enchantment, which only goes up to level 2.