problem
I found the option "Mass Enchantment Books" has little effect on gameplay, which means you can still make an op item when it's false through following process:

1. merge enchant books on vanilla anvil (up to repaircost limit 39)
2. prepare 8 such books
3. apply them on item by performing ritual

I know lowering the value "Mass Enchantment Max Level" will reduce the impact, but that applies to ALL enchant in game, which still causes limiting or weird behavior for some enchants regardless of its value.

solution
Allowing only enchanted books without RepairCost nbt to be used in rituals may solve this problem, since it guarantees books have never been merged on anvil.

alternatives
Tweaking "Mass Enchantment Max Level' to apply separately to each enchant in the game would be good, but then configuration takes too much time, which is not good. (especially on modpack with lots of enchantments)

I mean, this is exactly the reason "Mass Enchantment Max Level" is a thing. The purpose of the ritual is to surpass the boundaries of what vanilla allows in terms of enchanting. A "here, you've reached the end game, go nuts" kind of thing.
But I suppose it wouldn't hurt to add an option for limiting "tampered" books, so you're limited to what vanilla (or other mods) offer.

Actually when I said op item I was referring to an instant-game-crashing monstrosity with all enchant level 10 haha, but whatever, I see. Thank you for quick reply!