Does anyone know of a way to disable the adventure module on Forge without breaking the game? I can't switch to NeoForge since it causes large performance drops and TPS issues for some reason (working on a Modpack).

No, you have to go port this PR to legacy forge neoforged/NeoForge#115

Also there is absolutely no way you are seeing perf regressions caused simply by using NF, especially for 1.20.1, as it is effectively the exact same.