Deadlock during generation
BrainStone opened this issue ยท 4 comments
So I've managed to consitently run into a hard deadlock that freezes the server so hard not event the Paper Watchdog mercy kills the server. The server doesn't react to anything anymore and needs to be hardkilled in that state.
This only happens when using this plugin to pregenerate chunks. It typically happens 1-2 hours into the generation which has made it hard for me to further debug.
I suspect there may be some concurrent read/write with another plugin.
Other plugins that I believe might be involved in that deadlock are Dynmap, LandLord, WorldGuard (and Dynmap addons for those two). However I can't say definatlely.
Versions:
Paper: git-Paper-434 (MC: 1.16.5)
Chunky: 1.2.29
Dynmap: 3.1-beta6
LandLord: 4.340
WorldGuard: 7.0.4
Would you be able to check your logs to see if there is any information there that might help? In particular, if there is a specific chunk causing the issue, the processed chunk count would be the same or close in both logs, assuming you selected the same region both times.
Also, perhaps try without the other plugins installed (since they shouldn't be necessary while generating the world) and see if you still run into any issues.
It's a live server, they are very much necessary. Best I could do is without dynmap, but the other two are required.
And no there's nothing in the log. It just stops. Also I later continue the pregeneration and it freezes on another chunk.
I'll play around a little to see if I can find out any more info on my test server. I'm sorry for the very little details. It's just that it always either kicks players or prevents staff from working so I stopped trying to pregenerate.
Thank you for continuing to look into the issue. I understand that this is a bit tricky to debug so I am very appreciative that you are willing to look into it, and I will try to help you in any way I can to pinpoint what may be causing it.
You can try without dynmap, else I'd recommend downloading the server to run locally so that you can do so without the other plugins and players online. Pre-generation is typically not done on a live server (even though it does work without too much lag in 1.13+).
If you are seeing nothing in console, do you have update messages disabled (with the silent
command)? There should always be progress displayed in the logs which looks something like this:
[12:40:00 INFO]: [Chunky] Task running for world. Processed: 233 chunks (5.51%), ETA: 0:01:08, Rate: 57.9 cps, Current: 8, 0
You are able to see the current processed chunk count, and the current chunk being processed. That's why it can be very useful to check in the case that a particular chunk may be causing an issue. In particular, chunk loading issues are far more common with old maps compared to freshly generated ones, and if my guess is correct you are generating an old map to display on dynmap.
Although, as far as chunk loading issues go, usually you see them because a chunk is corrupt, and it prints a stack trace. Your particular case is a bit strange, since the server is freezing.
Perhaps you should also take a look at your server memory to check for leaks. With plugins, it's possible that they could be storing data in memory for each chunk, that is not cleared, and after a few hours your server might begin to hang because it doesn't have enough to operate normally. Spark's heap summary command might be able to point out a culprit if this is the case.
BTW in the case that you wish to try a different pre-generation tool to see if the issue is still occurring there vs Chunky (assuming you are able to replicate this behavior on your test server), please try WorldBorder. This has the best chance of producing any different results. Of course, I'd rather fix the issue here if there is one, however trying an alternative is often an easy debug option.