Minecraft version: 1.12.2
Wizardry version: 4.1.2

Issue details: Since Roguelike Dungeons does not support loot functions, maybe spell books can be added to them by making random spell book items that drop a random spell book when right clicked. The item should have variations that only drop a certain tier to make it balanced based on where I put it.

You can add loot tables (which can of course contain loot functions) to roguelike dungeons' chests: https://github.com/Greymerk/minecraft-roguelike/wiki/Loot-Rules#loot-table. Hopefully that should allow you to achieve the result you're looking for.