In vanilla, if you use an overhigh level enchanted book (like protection 5 > 4) on anvil, the game will treat it as max level. But quark can make those overhigh level book to use on anvil unrestrictedly.

1. Make sure that "Ancient Tome" is enabled in the config file.
2. Run command /give @s minecraft:diamond_helmet{RepairCost:34} /give @s minecraft:enchanted_book{StoredEnchantments:[{id:"minecraft:protection",lvl:4}]} /give @s minecraft:enchanted_book{StoredEnchantments:[{id:"minecraft:protection",lvl:5}]}
3. In vanilla, apply these 2 books to that diamond helmet, both of them will get a diamond helmet with protection 4 and cost 38 levels experience.
4. If quark is installed and ancient tome is enabled, protection 4 book works like vanilla, but protection 5 book will apply protection 5 on the helmet and only cost 10 levels experience, no matter how much RepairCost is.

Minecraft 1.16.5 Forge 36.2.35 Quark r2.4-322

I'm not sure what the issue is you're describing here? If you are giving yourself a book with commands it will always bypass any restrictions as nbt is not validated. i.e. You could reproduce the last step without even having Quark installed

I'm not sure what the issue is you're describing here? If you are giving yourself a book with commands it will always bypass any restrictions as nbt is not validated. i.e. You could reproduce the last step without even having Quark installed

You may misunderstand. In vanilla, enchanted book are always restricted. You CANNOT use enchanted book with overhigh level, no matter it's from command or another way, (diamond sword + unbreaking IV = diamond sword with unbreaking III). But if you install Quark, you CAN use overhigh level enchanted book unexpectedly.(diamond sword + unbreaking IV = diamond sword with unbreaking IV).

I mean that's one of the ancient tomes uses, to make enchanted books with a higher level than normal, you need to be able to combine an item with a over-enchanted book to get an over-enchanted item

I mean that's one of the ancient tomes uses, to make enchanted books with a higher level than normal, you need to be able to combine an item with a over-enchanted book to get an over-enchanted item

Ancient tomes can only make enchanted items have the higtest level+1 (diamond sword with unbreaking 3 + ancient tome with unbreaking 3 = diamond sword with unbreaking 4). That is in normal circumstances.
But this bug means you don't need ancient tome to get enchanted items with overhigh levels, what's more, its level are not restricted. In vanilla, an overhigh level enchanted book can only apply the highest level to the items, but if quark is installed, an enchanted book with unbreaking 255 can be applied to an item an the anvil, that's the bug.

How do you expect to get books higher then the max level in vanilla? quark doesnt magically let those books spawn, it only raises the max limit so tomes work

How do you expect to get books higher then the max level in vanilla? quark doesnt magically let those books spawn, it only raises the max limit so tomes work

Tomes do works as expected. But quark removed the restriction of enchanted book unexpectly. Any level of enchanted book can be used on anvil.
If you follow the steps at the top once, you'll understand.