Lag spikes on saving: Completely rework the background system for lighting!
mskurnik opened this issue ยท 16 comments
We are experiencing issue with lag spikes on my server and they seem to be related to the saveToDisk() method in the DiskIO class. We have about 50 flood lights in chunk loaded areas around the server. After hours of investigation with VisualVM and finally finding the sampler mod that lets you log only lag spikes, I've determined that floodlights is the likely culprit as it spikes from 3000-5000ms every 45-60 seconds. What is odd is the LightHandler.sav file is 870KB. While I don't know anything about Java development for Minecraft (I am a .NET C# developer) I was thinking you could modify that code to execute the save in a separate thread instead of using the Server thread.
Please let me know if you need anything other than the version numbers listed below.
- FloodLights: 0.6.8-96
- Forge Mod Loader - 7.10.99.99
- Minecraft Forge - 10.13.4.1492
This is weird, gonna look into that, never noticed that before.
Weirder is that there should be no lag spikes at all using the current code. I had almost 500 floodlights on my private server and it used about 0.5ms tick time permanently without lagging.
Also: 870kb for 50 floodlights is fairly large.
I am going to try to use my world from SMP with the 500 lights on a local server to see if it lags as well.
Yeah it seemed odd. I am not sure if you know of any issues with other mods or not but below is a list of mods that I have on my server. I can also send you my sav file if you have a way to read it outside of the game as it seems to just be your LightHandler object stored as a file so I have no way of reading it outside of a Java IDE.
[1.7.2] EnderBook v2.1.2.6.jar
[1.7.2+] KingCore v3.4.2.19.jar
ae2stuff-0.4.3.47-mc1.7.10.jar
AOBD-2.8.2.jar
appliedenergistics2-rv2-stable-10.jar
Aroma1997Core-1.7.10-1.0.2.15.jar
Aroma1997s-Dimensional-World-1.7.10-1.1.0.1.jar
ATG-1.7.10-0.12.0.jar
AutoSapling-1.1.jar
bdlib-1.9.1.97-mc1.7.10.jar
BiblioCraft[v1.10.6][MC1.7.10].jar
BigDoors-1.7.10-1.0.0.jar
BigReactors-0.4.3A.jar
buildcraft-7.0.22.jar
buildcraft-compat-7.0.11.jar
Carpenter's Blocks v3.3.7 - MC 1.7.10.jar
CodeChickenCore-1.7.10-1.0.7.47-universal.jar
CoFHCore-[1.7.10]3.0.3-303.jar
compacter-1.0.0.10-mc1.7.10.jar
ComputerCraft1.74.jar
Decocraft-2.1.1_1.7.10.jar
Dynmap-2.2-forge-1.7.10.jar
EnderCore-1.7.10-0.1.0.23_beta.jar
EnderStorage-1.7.10-1.4.7.36-universal.jar
EnhancedPortals_1.7.10-universal-3.0.12.jar
extrabiomesxl_1.7.10-3.16.2.jar
ExtraCells-1.7.10-2.2.73b129.jar
FloodLights-1.7.10-0.6.8-96.jar
forgeessentials-1.7.10-1.4.3.925-server.jar
ForgeMultipart-1.7.10-1.2.0.345-universal.jar
GalacticraftCore-1.7-3.0.12.350.jar
Galacticraft-Planets-1.7-3.0.12.350.jar
GardenStuff-1.7.10-1.6.3.jar
generators-0.9.17.102-mc1.7.10.jar
GrimoireOfGaia3-1.7.10-1.1.2.jar
ImmersiveEngineering-0.5.4.jar
immersiveintegration-0.5.2.jar
InventoryTweaks-1.59-dev-152.jar
ironchest-1.7.10-6.0.62.742-universal.jar
LunatriusCore-1.7.10-1.1.2.21-universal.jar
malisiscore-1.7.10-0.12.9.jar
malisisdoors-1.7.10-1.9.5.jar
Mantle-1.7.10-0.3.2a.jar
[MC1.7.10]deconstruction[v2.2.5].jar
mcjtylib-1.4.0.jar
MicdoodleCore-1.7-3.0.12.350.jar
Minecraft.ID.Resolver.v1.0.9.jar
MineFactoryReloaded-[1.7.10]2.8.0-104.jar
MobiusCore-1.2.5_1.7.10.jar
More-Planets-1.7.10-1.2.9.jar
MrCrayfishFurnitureModv3.4.8(1.7.10).jar
natura-1.7.10-2.2.0.1.jar
neiaddons-1.12.11.36-mc1.7.10.jar
NEIIntegration-MC1.7.10-1.0.12.jar
NetherOres-[1.7.10]2.3.0-12.jar
NotEnoughItems-1.7.10-1.0.5.111-universal.jar
ObsidianBoat-1.7.10-1.3.2.jar
OpenBlocks-1.7.10-1.4.4.jar
OpenModsLib-1.7.10-0.8.jar
OpenModularTurrets-1.7.10-2.1.4-182.jar
Opis-1.2.5_1.7.10.jar
OreDictionaryConverter-1.6.1.jar
Pam's HarvestCraft 1.7.10j.jar
Peaceful Plus [1.7.10].jar
Permissions.cfg
PlayerAPI-1.7.10-1.3.jar
QuartzPlus-1.7.10-0.9.04.jar
Railcraft_1.7.10-9.7.0.0.jar
RenderPlayerAPI-1.7.10-1.4.jar
Ruins-1.7.10.jar
sampler-1.45.jar
Schematica-1.7.10-1.7.6.128-universal.jar
Simple Recipes 1.7.10-1.8.2.jar
SimplyJetpacks-MC1.7.10-1.5.1.jar
SmartMoving-1.7.10-15.6.jar
SmartRender-1.7.10-2.1.jar
SolarFlux-1.7.10-0.8b.jar
spatialiocompat-build-9.jar
TConstruct-1.7.10-1.8.7.jar
thermalcasting-0.0.3.jar
ThermalDynamics-[1.7.10]1.1.0-161.jar
ThermalExpansion-[1.7.10]4.0.3B1-218.jar
ThermalFoundation-[1.7.10]1.2.0-102.jar
ThermalSmeltery-1.7.10-1.2.1.jar
TMechworks-1.7.10-0.2.14.100.jar
TorchLeversV1.4.1.jar
twilightforest-1.7.10-2.3.7.jar
Waila-1.5.10_1.7.10.jar
WailaHarvestability-mc1.7.x-1.1.2.jar
WAILAPlugins-MC1.7.10-0.1.2-21.jar
Wawla-1.3.1-1.7.10.jar
weaponmod-1.14.3.jar
worldedit-forge-mc1.7.10-6.0.2-SNAPSHOT-dist.jar
I think the issue is just because I have an odd, overengineered system. Gonna make the lighting system completely revamped in version 1.0.0. Noticed my testing world also hangs for a few seconds on saving.
In the meantime, do you think if I shut off all of the lights, let all the invisible light blocks despawn, shut down the server, and then delete the sav file it would at least improve my situation?
You can use a slightly older version that has the bugged system where the lights do not get saved properly, just do not try to modify the lighting during that time as it might rarely produce bugged lights with that versions.
Update @mskurnik if you want you can beta test the new lighting system here: https://www.ironhide.de/jenkins/job/FloodLights-dev/23/
It should result in no random lagspikes anymore. Though turning off or on a lot of lights at the same time will lag your client/server for a couple of seconds maybe. Also they are a lot more responsive.
Also you may need to turn all the lights off and on again.
OK. So I grabbed 1.0.0-23 off of Jenkins and my spikes immediately went away! Yay! However, another admin on the server decided to use a wrench to disable his lights by wrenching the thermal dynamics duct to the lights so they wouldn't be connected. I started to go around and reconnect the duct in creative mode and I noticed that a GUI opens up when I right click the floodlights with a duct. The GUI shows the power level and has a box that accepts an item. Being curious I placed an item in there. A few moments later the console started throwing errors and finally crashed. After booting the server back up and connecting it crashed again. Here are the exceptions:
Description: Ticking block entity
java.lang.NullPointerException: Ticking block entity
at de.keridos.floodlights.tileentity.TileEntityElectricFloodlight.narrowConeSource(TileEntityElectricFloodlight.java:201)
at de.keridos.floodlights.tileentity.TileEntityElectricFloodlight.addSource(TileEntityElectricFloodlight.java:381)
at de.keridos.floodlights.tileentity.TileEntityElectricFloodlight.func_145845_h(TileEntityElectricFloodlight.java:350)
at net.minecraft.world.World.func_72939_s(World.java:1939)
at net.minecraft.world.WorldServer.func_72939_s(WorldServer.java:489)
at net.minecraft.server.MinecraftServer.func_71190_q(MinecraftServer.java:636)
at net.minecraft.server.dedicated.DedicatedServer.func_71190_q(DedicatedServer.java:334)
at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:547)
at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:427)
at net.minecraft.server.MinecraftServer$2.run(MinecraftServer.java:685)
A detailed walkthrough of the error, its code path and all known details is as follows:
-- Head --
Stacktrace:
at de.keridos.floodlights.tileentity.TileEntityElectricFloodlight.narrowConeSource(TileEntityElectricFloodlight.java:201)
at de.keridos.floodlights.tileentity.TileEntityElectricFloodlight.addSource(TileEntityElectricFloodlight.java:381)
at de.keridos.floodlights.tileentity.TileEntityElectricFloodlight.func_145845_h(TileEntityElectricFloodlight.java:350)
-- Block entity being ticked --
Details:
Name: floodlights:electricFloodlight // de.keridos.floodlights.tileentity.TileEntityElectricFloodlight
Block type: ID #2059 (tile.floodlights:electricFloodlight // de.keridos.floodlights.block.BlockElectricFloodlight)
Block data value: 0 / 0x0 / 0b0000
Block location: World: (258,42,-4231), Chunk: (at 2,2,9 in 16,-265; contains blocks 256,0,-4240 to 271,255,-4225), Region: (0,-9; contains chunks 0,-288 to 31,-257, blocks 0,0,-4608 to 511,255,-4097)
Actual block type: ID #2059 (tile.floodlights:electricFloodlight // de.keridos.floodlights.block.BlockElectricFloodlight)
Actual block data value: 0 / 0x0 / 0b0000
Stacktrace:
at net.minecraft.world.World.func_72939_s(World.java:1939)
at net.minecraft.world.WorldServer.func_72939_s(WorldServer.java:489)
I am reverting to the backup prior to me putting something into the item slot and will reconnect the ducts once again. Hopefully it wont crash (as long as I don't put anything in there) and then you can work on disabling that GUI :). Thank you for being so responsive with this bug. It really sucked having the world freeze every 45-60 seconds.
Nope. It crashed again without ever once opening the GUI. I am going to revert back to the buggy version for now.
OK. I ended up using worldedit to replace the lights with sponge and then respawning them. The server seems a lot better now and the lights file is half of what it was before. I guess there was a glitched light.
The crash should absolutely not happen, I mean literally the crash that happened there was:
I create a block of light at the coordinates. then I grab the tileentity for that light from the exact same coordinates and try to run a command on it. This should NEVER even give a Nullpointerexception.
Weird stuff.
Can you try it with the newest version from the jenkins please, if the crash still occurs I need to add in a null check there. But I think that some Mod is interfering with the lights there.
I haven't gotten the crash since I removed and re-added the lights in a certain area. I think one of the lights was just bugged somehow.
Ah ok, so the new version works now?
Thanks you very much for the detailed analysis by the way.