I was having trouble with lag while playing All The Mods 3 mod pack so I ran VisualVM to see what was causing it. It appears ThaumicJEI Aspect Cache is eating up insane amounts of CPU time and memory.

edit: On the off chance someone else stumbles upon this thread after having similar problem: the easiest solution I found is to manually remove thaumicjei .jar-file from mods folder. As it's a client side mod removing it doesn't affect your ability to join servers running the same mod pack. The game runs noticeably faster after joining a game (and more importantly doesn't crash).

That's why there is a config option to disable the check every time you start the game.