Lately whenever im in a roguelike dungeon, and going down even deeper, I'm getting freezes when i notice the memory usage spike up to 100% each time, freezing for a very irritating amount of time, making it impossible to get through the dungeon in survival. I've been trying to fix this problem for a while, but nothing seems to work. I've tried allocating more memory to minecraft itself but no matter what, everytime i go into a roguelike dungeon the memory usage keeps spiking up. Setting the allocated memory to 4gb didn't work, 6gb didn't work. Each time it would just spike up to make up for the difference. I'm wondering if something else is wrong here or I have to allocate more memory, at the cost of stability. I'm not sure if this is a mod compatibility issue, or anything.

System:
Intel Core i5-6300HQ
GPU: Nvidia GTX 960M
16GB Memory installed

Any other information needed I will gladly provide.

This doesn't sound like a memory-related issue. I can run Rogue-like on less than 6gb of ram easily.
Perhaps your CPU/GPU is slacking behind?
Could you provide the java version you have installed?