Not sure if that's just the nature of the mod, but it's incredibly expensive, increasing the load times by like 30 seconds alone.

Is there anyway to remedy this?

Need more details about this. What mods? Patchouli by itself? Version?

Sorry, it was apparently only with other mods. I'm closing this.