If my camera is still, or I'm moving it fairly slowly, my framerate is consistently around 100-130 fps. However, if I move my mouse around fast, or even at a slightly higher than normal pace, it causes crazy lag spikes that are very annoying. I don't know what could be causing this, but I would like to have it fixed.

Do you experience lag spike when moving the mouse fast without any mod?

And Java usually needs JIT to improve performance but that process takes time. Try to play game normally for 5 minutes and then test the performance.

It didn't happen without mods, but for some reason I wasn't able to replicate it since I posted this even though it wasn't originally a one-off thing. Kind of strange, but I'm not complaining :)
Btw this is actually tied with optifine for my all time favorite mod, so thanks!