So with the new version of the mod, Fabric 1.20.4, the game will crash after the mods finished loading, here is the crash report:
Launcher: The game crashed whilst initializing game
Error: java.lang.RuntimeException: Mixin transformation of net.minecraft.class_355 failed

Latest.log file:
latest.log

This should be fixed across all versions now. While building the mod it just decided to randomly not create correct refmap files.