I find the Charms a bit too op, but I would like to keep the potions in the game, so disabling the Potion module all together isn't an option. Is there a way or could you add it?

You can remove the recipe for the charms and hide them from JEI, which should be sufficient for your goals.