Is there a simple way to do that? For example i'd like to set that enchanting anything on level, let's say 100 takes 25% of that levels so it would be 25 in this example. Right now i feel like enchanting is too op because of the fact that enchanting takes away basically no exp.

No, the cost is fixed at 1/2/3 levels from the individual slots. Changing that would require rewriting a reasonable portion of the menu, so this isn't something I will be exploring.