//Bad english incoming
When i apply looting 10 on a sword its the maximum amount there are books for in JEI, ok then i tried and combined looting 10 and looting 10 (books) in an anvil. and well i got an looting 11 book. "thooo..." i thought lets try two books looting 11 and looting 11. well nothing. just no valid recipe.
then i combined a looting 11 book and a sword, you guessed it: looting 11 sword. then i combined two looting 11 swords in an anvil. well ... looting 12 sword. what is that? it's just not fun

I fixed it i think.. kind of..
i'm glad to tell you its not your mod so this thread can be closed.
Its "Quark" (Mod)
In quark there is a book named Ancient Tome
https://ftb.gamepedia.com/Ancient_Tome
it allowes you to enchant a book one level higher as intended. But (so i think) destroyes everything obove.
Yes you are right i have installed a lot of mods wich are messing with enchantments, i am really sorry that i reduced the problem to your mod.
Thanks alot anyway!

Oh the minecraft V1.12.2 and a mod list isn't that easy i testet everything yesterday many hours it worked great when i disabled that Ancient Tome. It must be some kind of dirty programming there...

nice that you found the issue :)

are you sure this is caused by this mod?
i know about other mods that also mess with the max level of enchantments, and you seem to use atleast a few other mods.
Please tell me the minecraft version you are playing, the version of this mod, and a list of all the other used mods, so i can try and reproduce this.
also please try and reproduce this with as little mods as possible.