[BUG] Server hangs on shutdown

Question

[BUG] Server hangs on shutdown

MilkMC opened this issue 5 years ago · 45 comments

MilkMC commented 5 years ago

What happened?

When I use the command /stop soon after starting the server it says it saves all chunks, but then it "hangs" until 5 minutes have passed since the server started. If I wait at least 5 minutes and then do the command, the server shuts down normally.

Edit: Sometimes it may occur even after the server has been running for a long time. Exact circumstances to cause this are unknown.

What should happen?

The server should be able to shut down normally even after it just started. I know this is not a usual server use case, but this is a test server for a new 1.15.2 mod and I'm restarting often to make small adjustments.

Steps to reproduce

Run the mod version 1.1.13 with forge 31.1.27 for Minecraft 1.15.2
Do /stop right after the server starts.

Version

1.12
1.14
[✓] 1.15.2

Mod version 1.1.13

Config file

Default file, no changes were made.

Additional info

I've been using 1.7.10 for years but this is my first experience in 1.15 so I might just be doing something wrong.

Logs
This first set of logs is when I shutdown the server as soon as it has started.
debug-1.log
restart-1.txt

This second set of logs is when I wait 5 minutes or so before shutting down the server.
debug-2.log
restart-2.txt

This is the startup script I am using. File Extension changed from sh to txt for upload.
start.txt

McKitten · Answer 1 · 2020-06-24T18:28:47.000Z

Servers are running CentOS and getting this in the server output AFTER the server does it shutdown save on both 1.14.4 & 1.15.2 most current versions

https://pastebin.com/qx51GT4K

Knocks it out for about 6minutes without fail.
Wouldn't be looking for a library that isn't available publicly would it?

Tander · Answer 2 · 2020-06-24T21:26:41.000Z

Just for the record: my problem was irrelevant to this ticket. I updated mods in the pack a couple of weeks ago and it works without any issues now.

katubug · Answer 3 · 2020-07-11T22:16:44.000Z

I am having this issue as well on 1.15.2, Forge 31.2.30. DCIntegration 1.3.1.

I think this may also be related to my players timing out? Someone will get booted and then no one will be able to connect until I kill and restart the server. I've also found this in my log, usually at the same time the timeout happens:

>....Exception in thread "Thread-73" [15:10:20] [Thread-73/INFO] [STDERR/]: [java.lang.ThreadGroup:uncaughtException:1052]: java.lang.IllegalArgumentException: ID may not be null
>.... [15:10:20] [Thread-73/INFO] [STDERR/]: [java.lang.ThreadGroup:uncaughtException:1052]: at net.dv8tion.jda.internal.utils.Checks.notNull(Checks.java:56)
>.... [15:10:20] [Thread-73/INFO] [STDERR/]: [java.lang.ThreadGroup:uncaughtException:1052]: at net.dv8tion.jda.internal.utils.Checks.notEmpty(Checks.java:61)
>.... [15:10:20] [Thread-73/INFO] [STDERR/]: [java.lang.ThreadGroup:uncaughtException:1052]: at net.dv8tion.jda.api.utils.MiscUtil.parseSnowflake(MiscUtil.java:105)
>.... [15:10:20] [Thread-73/INFO] [STDERR/]: [java.lang.ThreadGroup:uncaughtException:1052]: at net.dv8tion.jda.api.utils.cache.SnowflakeCacheView.getElementById(SnowflakeCacheView.java:59)
>.... [15:10:20] [Thread-73/INFO] [STDERR/]: [java.lang.ThreadGroup:uncaughtException:1052]: at net.dv8tion.jda.api.JDA.getUserById(JDA.java:539)
>.... [15:10:20] [Thread-73/INFO] [STDERR/]: [java.lang.ThreadGroup:uncaughtException:1052]: at de.erdbeerbaerlp.dcintegration.DiscordIntegration.lambda$playerJoin$0(DiscordIntegration.java:170)
>.... [15:10:20] [Thread-73/INFO] [STDERR/]: [java.lang.ThreadGroup:uncaughtException:1052]: at java.lang.Thread.run(Thread.java:748)

ErdbeerbaerLP · Answer 4 · 2020-07-11T22:26:13.000Z

Oh, forgot an null check :/
Will fix that one in the next update. But that error is not relevant to the shutdown problem, as the code is newer than the issue

katubug · Answer 5 · 2020-07-11T22:28:48.000Z

Ah okay, good to know. Thanks! If I can get any more information to you about the shutdown issue, let me know! I'm happy to help out if I can.

katubug · Answer 6 · 2020-07-21T01:49:08.000Z

Just wanted to update for my case - the shutdown hang was caused by LagMeter. Removing that mod made my server able to shut down normally again. It does still stall out sometimes, though. Unsure if that's related.

LemADEC · Answer 7 · 2020-07-28T19:50:14.000Z

Server is hanging on shutdown just after this callstack while using version 1.3.2-1.15

[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:write:277]: [JDA MainWS-ReadThread] ERROR WebSocketClient - There was an error in the WebSocket connection
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:log:272]: java.lang.NoClassDefFoundError: net/dv8tion/jda/api/requests/CloseCode
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:log:272]:        at net.dv8tion.jda.internal.requests.WebSocketClient.onDisconnected(WebSocketClient.java:384)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:log:272]:        at dcshadow.com.neovisionaries.ws.client.ListenerManager.callOnDisconnected(ListenerManager.java:224)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:log:272]:        at dcshadow.com.neovisionaries.ws.client.WebSocket.finish(WebSocket.java:3730)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:log:272]:        at dcshadow.com.neovisionaries.ws.client.WebSocket.onThreadsFinished(WebSocket.java:3696)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:log:272]:        at dcshadow.com.neovisionaries.ws.client.WebSocket.onReadingThreadFinished(WebSocket.java:3663)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:log:272]:        at dcshadow.com.neovisionaries.ws.client.ReadingThread.notifyFinished(ReadingThread.java:1138)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:log:272]:        at dcshadow.com.neovisionaries.ws.client.ReadingThread.runMain(ReadingThread.java:80)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:log:272]:        at dcshadow.com.neovisionaries.ws.client.WebSocketThread.run(WebSocketThread.java:45)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:write:279]: Caused by: java.lang.ClassNotFoundException: net.dv8tion.jda.api.requests.CloseCode
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:write:279]:      at java.lang.ClassLoader.findClass(ClassLoader.java:523)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:write:279]:      at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:write:279]:      at cpw.mods.modlauncher.TransformingClassLoader.loadClass(TransformingClassLoader.java:101)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:write:279]:      at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
[05:35:43] [JDA MainWS-ReadThread/INFO] [STDERR]: [net.dv8tion.jda.internal.utils.SimpleLogger:write:279]:      ... 8 more

1.2.7-1.15#69 was showing a similar error at shutdown.

MrHarrie · Answer 8 · 2020-07-31T14:33:10.000Z

I have the same problem on ubunto 18.04.4 This seems to be a major issue on linux?

khloroshue · Answer 9 · 2020-08-10T23:19:54.000Z

Having the same issue here. Doesn't seem to matter how long the server has been running. Experiencing the same issue running on a VPS as well. My home machine encounters the issue and is using the following versions:

Windows 10 x64 1909
Java 1.8.0_261
Forge 1.16.1-32.0.108
DiscordIntegration v. 1.3.3 - 1.16.

ErdbeerbaerLP · Answer 10 · 2020-08-11T06:35:36.000Z

OK. Then it seems to not be an Linux specific issue. I just can't reproduce it on windows. 🤔

khloroshue · Answer 11 · 2020-08-14T20:56:02.000Z

I suspect this is an issue on Discord's end - I don't know if it's holding a socket open on their end or something, but I've tried several mods that hook into discord, and the issue exists on every mod I've tried. Maybe it's a result of rate limiting or something of the like that they've implemented to deal with load.

github-actions · Answer 12 · 2020-09-14T00:25:41.000Z

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

Ki-Tan2934 · Answer 13 · 2020-09-15T18:20:48.000Z

Was a fix ever found?

ErdbeerbaerLP · Answer 14 · 2020-09-16T09:17:24.000Z

Currently not, no :/

Ki-Tan2934 · Answer 15 · 2020-09-16T11:42:28.000Z

Maybe add a way for Rcon to force stop DC intergrations before /stop is used
For example /Dstop (kills discord bot) Wait 10s - /stop

Ki-Tan2934 · Answer 16 · 2020-09-16T11:44:24.000Z

Or implement a new command that kills the mod before the server closes could even be an additional mod called DCFIX-Linux

TymonU · Answer 17 · 2020-09-27T18:14:24.000Z

Same problem here. I think it has to do with checking for possible server crash. The discord plugin I am using from sponge on my old world only watches for the stop command.
Edit: for more info. server is running centos 8 and on the newest forge 34.1.0 with 1.16.3. Chat from server to discord also not working. different issue and will get the log for that one.

Fummowo · Answer 18 · 2020-04-26T02:14:13.000Z

I'm getting this exact same issue using Minecraft 1.14.4 with version 1.1.14 except it doesn't matter for how long the server has been running for and it will stay paused until I kill java. The issue only showed up as soon as I downloaded this mod, although it's a great mod, very nice job btw :). I am running Forge version 28.2.0 (Recommended).

github-actions · Answer 19 · 2020-05-27T00:16:33.000Z

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

MilkMC · Answer 20 · 2020-05-27T00:24:31.000Z

Can this get fixed?

ErdbeerbaerLP · Answer 21 · 2020-05-27T06:57:44.000Z

I am unable to reproduce this bug and the log files also seem to not show any errors. I only have one idea: disable the shutdown message by setting it blank (1.15/1.12 only). If it fixes that it might be caused by how I force the message to be sent before server actually stops

ErdbeerbaerLP · Answer 22 · 2020-05-27T10:35:38.000Z

I think I found it. I tested using Linux and it had the delay

MilkMC · Answer 23 · 2020-05-27T09:09:41.000Z

I changed this line in the config to be as shown:

msgServerStopped = ""

But I get a crash saying that it can't be empty. Interestingly, the server still hangs after it crashes which is not the behavior I would expect, but perhaps this can help narrow down the cause.
crash-2020-05-27_08.38.03-server.txt

Tander · Answer 24 · 2020-05-31T13:26:19.000Z

Oh, actually, I have the same problem on my server. Removing this mod helped. Ping me if you need any information about my setup.

forge-31.2.0
dcintegration-1.1.14-1.15 (today updated to 1.2.4, let's see if it helps)

UPD: No, 1.2.5 still have this issue.
Here how it looks (server were running for at least 7 hours before shutdown):

> say SERVER REBOOT IN 10 SECONDS!
[16:07:01] [Server thread/INFO] [minecraft/DedicatedServer]: [Server] SERVER REBOOT IN 10 SECONDS!
> stop
[16:07:11] [Server thread/INFO] [minecraft/DedicatedServer]: Stopping the server
[16:07:11] [Server thread/INFO] [co.sk.wo.ex.pl.PlatformManager/]: Unregistering com.sk89q.worldedit.forge.ForgePlatform from WorldEdit
[16:07:11] [Server thread/WARN] [environmentalcreepers/]: Invalid entity class name (class not found): 'appeng.entity.EntitySingularity'
[16:07:11] [Server thread/WARN] [environmentalcreepers/]: Invalid explosion class name (class not found): 'slimeknights.tconstruct.gadgets.entity.ExplosionEFLN'
[16:07:11] [Server thread/INFO] [thutessentials/]: Server Stopped
[16:07:11] [Server thread/INFO] [simplelogin/]: Shutting down player login handler
[16:07:12] [Server thread/INFO] [simplelogin/]: Saving all entries
[16:07:12] [Server thread/INFO] [minecraft/MinecraftServer]: Stopping server
[16:07:12] [Server thread/INFO] [minecraft/MinecraftServer]: Saving players
[16:07:12] [Server thread/INFO] [minecraft/MinecraftServer]: Saving worlds
[16:07:12] [Server thread/INFO] [minecraft/MinecraftServer]: Saving chunks for level 'world'/minecraft:overworld
[16:07:12] [Server thread/INFO] [minecraft/ChunkManager]: ThreadedAnvilChunkStorage (world): All chunks are saved
[16:07:12] [Server thread/INFO] [minecraft/ChunkManager]: ThreadedAnvilChunkStorage (world): All chunks are saved

And any command after that does nothing. Last message in discord was Server Stopped!

Tander · Answer 25 · 2020-06-01T14:41:19.000Z

A-ha! I think I caught that. botModifyDescription = false fixed the issue for me it seems. But i'm not sure, need more testing.

MilkMC · Answer 26 · 2020-06-01T16:30:12.000Z

That didn't seem to fix it for me. But if you have a working config, please send it (excluding your bot token and channel).

Tander · Answer 27 · 2020-06-01T17:03:25.000Z

https://pastebin.com/M3Q3CeXm (it's from my recent bugreport #72 )
Just change botModifyDescription to false in it.

Tander · Answer 28 · 2020-06-01T20:15:03.000Z

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.4 LTS
Release:	18.04
Codename:	bionic

$ java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

You may find my list of mods in crashlog in #72

MilkMC · Answer 29 · 2020-06-01T17:51:47.000Z

I copied your config, changed that line to false, and updated to version 1.2.5 but it's still hanging on shutdown.

ErdbeerbaerLP · Answer 30 · 2020-06-01T17:54:17.000Z

It is some weird issue with Linux.
Testing using windows cmd: no problems.
Testing using windows subsystem for linux (Ubuntu) I get that.
Not tested on other Linux versions tho

MilkMC · Answer 31 · 2020-06-01T18:18:28.000Z

Yeah, I'm on Ubuntu 18.04.4 LTS.

Tander · Answer 32 · 2020-06-02T14:02:25.000Z

Yes, can confirm: botModifyDescription = false fixed this issue for me. Server just made full cycle (stop - start - backup - working for 12 hours - stop) without any actions from my side. So probably my issue wasn't a part of MilkMC's one, just similar symptoms.

Since botModifyDescription helped for me, I guess my case was a part of #70 instead? Still hope it will be fixed, I liked botModifyDescription feature.

ErdbeerbaerLP · Answer 33 · 2020-06-02T14:03:41.000Z

Maybe they change it back when this post gets enough upvotes.
https://support.discord.com/hc/en-us/community/posts/360067755532-Increase-rate-limit-for-editing-channel-description

Tander · Answer 34 · 2020-06-03T06:58:34.000Z

And it hung up again. But with new errors this time.
https://pastebin.com/U29kLt7f

ErdbeerbaerLP · Answer 35 · 2020-06-03T06:59:07.000Z

That is a different mod

Tander · Answer 36 · 2020-06-03T07:10:20.000Z

Yeah, but it's weird: I haven't update this mod when my issue was started, and new error says that is something wrong with config, but I haven't changed it either.

Ki-Tan2934 · Answer 37 · 2020-10-08T20:07:51.000Z

Was there ever a fix? 1.16.3 does the same all I really want is discord chat and nothing more
1.12.2 Minecraft does not have a problem

Ki-Tan2934 · Answer 38 · 2020-10-09T00:37:54.000Z

Sounds good

ErdbeerbaerLP · Answer 39 · 2020-10-08T20:56:14.000Z

Finally got some time, so I planned to do an half-rewrite of this mod to add new features and fix bugs. Maybe this bug vanishes too then. Also leak I plan on doing an spigot port of this mod.

About that rewrite thing, I plan on completely rework the Mod main class and the Discord class, which contains all the Discord specific interactions like sendMessage methods

Ki-Tan2934 · Answer 40 · 2020-10-10T13:30:29.000Z

I can test on Linux if you want

Ki-Tan2934 · Answer 41 · 2020-10-10T14:50:48.000Z

Could you add a /discord stop command

Ki-Tan2934 · Answer 42 · 2020-10-10T20:53:19.000Z

I did manage to get a crashlog don't think it'll help https://paste.ubuntu.com/p/rwwHCksSKF/

To crash just do /discord restart then /stop right after

koderian · Answer 43 · 2020-10-11T21:46:01.000Z

Same problem, I just opened a issue with some details

sfPlayer1 · Answer 44 · 2020-10-31T23:04:11.000Z

I noticed a couple non-daemon okhttp threads, which may be the root cause behind this. While those threads may have a way to be orderly terminated, this wouldn't be possible in a crash.

square/okhttp#5834 fixes this original issue, but JDA itself is still on okhttp 3.13 and would need to update...

ErdbeerbaerLP · Answer 45 · 2020-11-01T16:32:31.000Z

Ok, will include the new okhttp version in the next update. Just tested replacing the version of JDA with the latest one and it works AND fixes this problem!

Share to

Version