This is potentially not a problem on your end but here goes:

I tried to add the latest version to my 1.16.5 modpack. but after testing a bit, I realize that the world would suffer from freezes much more frequently without world-gen. So I used a mod called spark to locate the things that are taking up resources, surprisingly, a skeleton mob with the specialAI is taking up a whooping 16% of the resources. Here's the log with the highlighted parts indicating that this mod is the one causing the issue: https://spark.lucko.me/JyoiwXMBWi?hl=107,66

I also tried removing this mod and playing on other worlds, the issue did not occur.

Is there a way to fix this? I really like the concept and I think it can help make the game a LOT more interesting, thanks in advance!

Looking into this, it looks like it is caused specifically by the idle fiddling AI attempting to interact with something. I am not sure what exactly is going on here since it shouldn't be spamming this all that frequently and blocks to interact with are very rarely included in world gen. Perhaps creating the fake player instance needed to interact just has a crazy overhead.

For a temporary fix, I recommend disabling the idle fiddling AI (remove all mobs from idle_fiddling.entities.whitelist so no new mobs are given the AI) or impose some restrictions on the AI, such as removing all fiddling targets that can appear in world gen. I will try to address the issue in the next SAI version.

Should be fixed in most recent release