When attempting to set up a server with the following mods I was getting netty errors that always ended with out of memory errors related to permgen. This was extremely hard to track down since the server doesn't always hang right away. There is never a crash rather the server will hang until it is force restarted. I'm not sure what other mod on this list is causing the conflict if it is in fact a conflict but I do know that removing railcraft fixes the issue (at least it is for now). I believe this to be a mod conflict because setting up another server with just railcraft the issue doesn't seem to ever occur.

BiblioCraft[v1.7.5][MC1.7.10].jar
Blood Magic-v1.1.0(1.7.10).jar
buildcraft-6.0.17.jar
Carpenter's Blocks v3.2.6 - MC 1.7.10.zip
CodeChickenCore-1.7.10-1.0.2.9-universal.jar
CoFHCore-[1.7.10]3.0.0B5-30.jar
DeathCounter-4.0.0.jar
EnderIO-1.7.10-2.0.1.194.jar
extrautilities-1.1.0i.jar
Forbidden Magic-1.7.10-0.5.jar
forestry_1.7.10-3.0.0.90.jar
IguanaTinkerTweaks-1.7.10-2.0.1B2.jar
immibis-core-59.0.1.jar
ironchest-1.7.10-6.0.49.733-universal.jar
Jabba-1.1.4_1.7.10.jar
liquid-xp-59.0.0.jar
Mantle_mc1.7.10_0.3.0.jar
MineFactoryReloaded-[1.7.10]2.8.0RC2-557.jar
Morpheus-1.7.10-1.5.18.jar
natura-1.7.10-2.2.0-b1.jar
NotEnoughItems-1.7.10-1.0.2.15-universal.jar
OpenEye-0.6-1.7.10.jar
Pam's HarvestCraft 1.7.10a.zip
ProjectRed-Base-1.7.10-4.4.5.44.jar
ProjectRed-Compat-1.7.10-4.4.5.44.jar
ProjectRed-Integration-1.7.10-4.4.5.44.jar
ProjectRed-Lighting-1.7.10-4.4.5.44.jar
ProjectRed-Mechanical-1.7.10-4.4.5.44.jar
ProjectRed-World-1.7.10-4.4.5.44.jar
Railcraft_1.7.10-9.2.2.0.jar
secretroomsmod-1.7.10-4.7.1.372.jar
StevesCarts2.0.0.b16.jar
TConstruct_mc1.7.10_1.6.0d39.jar
Thaumcraft-1.7.10-4.2.0.0.jar
ThaumcraftMobAspects-1.7.x-2B-snapshot-17.jar
thaumicenergistics-0.6.2a.jar
ThaumicExploration-1.7.10-1.17.jar
ThaumicTinkerer-2.5-1.7.10-154.jar
ThermalExpansion-[1.7.10]4.0.0B5-13.jar
ThermalFoundation-[1.7.10]1.0.0B3-8.jar
TMechworks_mc1.7.10_0.2.8.jar
twilightforest-1.7.2-2.2.3.jar
witchery-1.7.10-0.20.5.jar

I've also posted this on the forum here.

If you are having permgen errors, increase your permagen.

This is not what you think it is considering I can give the sever 2GB of permgen space and this issue still occurs. This is a meoryleak, it's not simply a matter of giving the server more memory because it doesn't matter how much memory I give it.

You'd be better off posting the log than trying to guess. It is nearly impossible to create a memory leak in Permgen space unless you are dynamically creating classes, and I'm pretty sure no MC mod has any reason to do that.

The issue eludes me it seems removing any single mod that adds a lot of content fixes the issue however increasing the permgen by any amount doesn't fix the issue. That said it doesn't seem to be a railcraft issue rather railcraft was the last mod I added to make it occur in my tests. Not sure how to resolve this so for now I'm just going to remove one of the mods.