Severe Server Lag
xinyuan-liu opened this issue · 21 comments
I am running a FTB Interactions 1.0.0 server with AstralSorcery 1.10.12 on a high-end computer with 8700K CPU and 16GB RAM.
My server encountered severe lag problem on even early games. I assume this lag is caused by AstralSorcery.
Here are some references for you:
Each time after GameStages saved stages for players, the server will stall for up to 20 seconds:
And the average TPS is healthy:
Here are the profile result from Sampler mod:
And the complete nps file:
sampler-snapshot_1.zip
I assume the doSave method is stalling the server and wonder why this happens.
Inside your world folder (where the folders 'region', 'stats', 'playerdata', and so on) are, there should be a folder called "AstralSorceryData" - would you be so kind as to .zip package it and upload somewhere?
Thanks in advance.
AstralSorceryData.zip
Here it is.
Also encountered a similar problem.
Timings: https://timings.aikar.co/?id=be9e80f79c464e2dada9870e33506177
"Watchdog":
Thread: Server thread, state: RUNNABLE
at java.io.FileOutputStream.write(FileOutputStream.java)
at java.io.FileOutputStream.write(FileOutputStream.java:290)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:200)
at net.minecraft.nbt.NBTTagInt.func_74734_a(SourceFile:25)
at net.minecraft.nbt.NBTTagCompound.func_150298_a(NBTTagCompound.java:474)
at net.minecraft.nbt.NBTTagCompound.func_74734_a(NBTTagCompound.java:35)
at net.minecraft.nbt.NBTTagList.func_74734_a(NBTTagList.java:34)
at net.minecraft.nbt.NBTTagCompound.func_150298_a(NBTTagCompound.java:474)
at net.minecraft.nbt.NBTTagCompound.func_74734_a(NBTTagCompound.java:35)
at net.minecraft.nbt.CompressedStreamTools.func_150663_a(CompressedStreamTools.java:112)
at net.minecraft.nbt.CompressedStreamTools.func_74800_a(CompressedStreamTools.java:102)
at net.minecraft.nbt.CompressedStreamTools.func_74795_b(CompressedStreamTools.java:152)
at hellfirepvp.astralsorcery.common.data.world.WorldCacheManager.saveDataToFile(WorldCacheManager.java:203)
at hellfirepvp.astralsorcery.common.data.world.WorldCacheManager.doSave(WorldCacheManager.java:230)
at hellfirepvp.astralsorcery.common.event.listener.EventHandlerIO.onSave(EventHandlerIO.java:58)
at net.minecraftforge.fml.common.eventhandler.ASMEventHandler_1162_EventHandlerIO_onSave_Save.invoke(.dynamic)
at net.minecraftforge.fml.common.eventhandler.ASMEventHandler.invoke(ASMEventHandler.java:90)
at net.minecraftforge.fml.common.eventhandler.EventBus.post(EventBus.java:746)
at org.spongepowered.mod.event.SpongeToForgeEventFactory.createAndPostWorldSaveEvent(SpongeToForgeEventFactory.java:936)
at org.spongepowered.mod.event.SpongeToForgeEventFactory.createAndPostForgeEvent(SpongeToForgeEventFactory.java:287)
at org.spongepowered.mod.event.SpongeModEventManager.post(SpongeModEventManager.java:338)
at org.spongepowered.mod.event.SpongeModEventManager.extendedPost(SpongeModEventManager.java:435)
at org.spongepowered.mod.event.SpongeModEventManager.post(SpongeModEventManager.java:404)
at org.spongepowered.common.event.SpongeEventManager.post(SpongeEventManager.java:465)
at net.minecraft.world.WorldServer.func_73044_a(WorldServer.java:2472)
at org.spongepowered.common.world.WorldManager.saveWorld(WorldManager.java:540)
at net.minecraft.server.MinecraftServer.func_71267_a(MinecraftServer.java:3884)
at net.minecraft.server.MinecraftServer.constant$getSaveTickInterval$zjn000(MinecraftServer.java:3837)
at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:687)
at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:526)
at java.lang.Thread.run(Thread.java:748)
@SSKirillSS does it happen without Sponge involved?
I've read the code of WorldCacheManager and RockCrystalBuffer. it seems that you store all the locations of rock crystals in a whole dimension in a single file. This file may become very large when the map is expansive.
In my case, an single 2.6MB nbt file is stored in AstralSorceryData/DIM_100 and it is too heavy for Minecraft to save in a singe time and will stall the server.
I suggest that you shall store the locations of rock crystals in several files separated by chunks or regions and save only the dirty ones each time and/or use some other approaches to save the locations.
@Doomgull Yes it does happen without spongeforge. Xinyuan doesnt have spongeforge as you can see in his profiler.
I asked SSKrillSS because his explicitly had sponge, and wanted to verify if it was same issue or something else.
A possible workaround for this (if it is making servers unplayable) would be to mount a ramdisk and use that for the folder structure that AS is using.
It goes without saying this could be unstable.
A possible workaround for this (if it is making servers unplayable) would be to mount a ramdisk and use that for the folder structure that AS is using.
It goes without saying this could be unstable.
Yeah, I've tried ram disk, but still huge lag. 2.6MB NBT data is too heavy for MC to store. The bottle neck is not the disk IO but the serialization process on CPU.
Moving all data from a per-world storage to chunk capabilities.
Next version will have a command ("/as migrate-data") which will transfer all rock crystal data to those chunks. Keep in mind that that might be a VERY intensive operation so better do that through console while noone is online!
Hello! Thank you so much for fixing the issue. Once you release the new version would us, server owners be able to update the mod server-side only? Or clients need to have it too?
I will include other stuff in there aswell, so i'm afraid the clients would also need to update. :/
ETA is: When it comes out.
If you want immediate notification, join the discord and watch the changelog channel.
Alright, so the update 1.10.13 includes the command /as migrate-data which will convert all the rock crystal data from the "old" per-world format to storing them per chunk making it overall much more performant.
Keep in mind to make backups of your worlds before doing that! Also keep in mind that it might take a while considering you probably have generated a MASSIVE world with tons of chunks and stuff that need to be processed.
Keep me informed about the result if you don't mind.
