server crash!
netherstar opened this issue · 26 comments
sometime i have this issues and server crash,
http://paste.ubuntu.com/16629026/
Can you check this?
@games647 can you check this? sometimes huge amount of players is disconnect
spigot show this:
luantran159 lost connection: Internal Exception: java.io.IOException: Connection reset by peer
bungeecord show this:
20:19:47 [SEVERE] [/14.185.202.219:49493|QuangSon1] <-> DownstreamBridge <-> [medieval-factions] - encountered exception
java.lang.IllegalArgumentException: Objective Stats already exists in this scoreboard
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at net.md_5.bungee.api.score.Scoreboard.addObjective(Scoreboard.java:55)
at net.md_5.bungee.connection.DownstreamBridge.handle(DownstreamBridge.java:126)
at net.md_5.bungee.protocol.packet.ScoreboardObjective.handle(ScoreboardObjective.java:54)
at net.md_5.bungee.netty.HandlerBoss.channelRead(HandlerBoss.java:77)
at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:154)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:354)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:145)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:154)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:354)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:145)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:154)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:354)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:145)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:277)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:372)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:245)
at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:154)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:354)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:145)
at io.netty.handler.timeout.ReadTimeoutHandler.channelRead(ReadTimeoutHandler.java:152)
at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:154)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:354)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:145)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:1078)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:527)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:484)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:398)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:370)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
at java.lang.Thread.run(Thread.java:745)
and sometimes server crash with this error:
[20:40:36] [Paper Watchdog Thread/ERROR]: Server thread dump (Look for plugins here before reporting to Paper!):
[20:40:36] [Paper Watchdog Thread/ERROR]: ------------------------------
[20:40:36] [Paper Watchdog Thread/ERROR]: Current Thread: Server thread
[20:40:36] [Paper Watchdog Thread/ERROR]: PID: 18 | Suspended: false | Native: false | State: RUNNABLE
[20:40:36] [Paper Watchdog Thread/ERROR]: Stack:
[20:40:36] [Paper Watchdog Thread/ERROR]: org.bukkit.craftbukkit.v1_9_R1.entity.CraftEntity.getEntityId(CraftEntity.java:275)
[20:40:36] [Paper Watchdog Thread/ERROR]: org.bukkit.craftbukkit.v1_9_R1.entity.CraftPlayer.equals(CraftPlayer.java:300)
[20:40:36] [Paper Watchdog Thread/ERROR]: java.util.WeakHashMap.eq(WeakHashMap.java:287)
[20:40:36] [Paper Watchdog Thread/ERROR]: java.util.WeakHashMap.get(WeakHashMap.java:401)
[20:40:36] [Paper Watchdog Thread/ERROR]: com.github.games647.scoreboardstats.scoreboard.protocol.PacketSbManager.getScoreboard(PacketSbManager.java:43)
[20:40:36] [Paper Watchdog Thread/ERROR]: com.github.games647.scoreboardstats.scoreboard.protocol.PacketSbManager.onUpdate(PacketSbManager.java:55)
[20:40:36] [Paper Watchdog Thread/ERROR]: com.github.games647.scoreboardstats.RefreshTask.run(RefreshTask.java:54)
[20:40:36] [Paper Watchdog Thread/ERROR]: org.bukkit.craftbukkit.v1_9_R1.scheduler.CraftTask.run(CraftTask.java:58)
[20:40:36] [Paper Watchdog Thread/ERROR]: org.bukkit.craftbukkit.v1_9_R1.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:352)
[20:40:36] [Paper Watchdog Thread/ERROR]: net.minecraft.server.v1_9_R1.MinecraftServer.D(MinecraftServer.java:822)
[20:40:36] [Paper Watchdog Thread/ERROR]: net.minecraft.server.v1_9_R1.DedicatedServer.D(DedicatedServer.java:404)
[20:40:36] [Paper Watchdog Thread/ERROR]: net.minecraft.server.v1_9_R1.MinecraftServer.C(MinecraftServer.java:723)
[20:40:36] [Paper Watchdog Thread/ERROR]: net.minecraft.server.v1_9_R1.MinecraftServer.run(MinecraftServer.java:622)
Are cause by ScoreboardStats?
The first one is a bungeecord bug, because it doesn't clear the scoreboard objective on server switch. Could you make a timings report for the seconds issue.
Second issue happen after first issue 5 minutes, and server crash, so i cant have timinga for you.
Maybe your watchdog timeout is too low. Could you check it please in the spigot configuration.
i dont known why, but this issues happen every afternoon (~ 15, 16 PM)
Can i ask, what is watchdog? i see spigot.yml, but can't find watchdog
I think it's a memory leak of another plugin, which holds a reference on the player object. ScoreboardStats holds only a weak reference. It means once it's no longer needed (i.e. disconnects), it will be deleted. When another plugin still uses it just holds it in memory, ScoreboardStats cannot delete it and the collection will be bigger and bigger.
There multiple options in the spigot.yml for the watchdog.
- timeout-time
- restart-on-crash
- restart-script
A Watchdog is security concept used in many applications. It will run separately from the main applications and checks if the application is still responding. If not, stop it or show a user notification.
It's not recommended to higher this value, because if the server is not responding for 60 seconds, something is really wrong. Although it might happen for big operations like worldediting.
it's mean, there are plugin has memory leak problem, which Scoreboardstats use variable of it?
My scoreboardstat use variable of:
- faction
- Vault eco
- mcmmo
- ping variable
can Other scoreboard plugin conflict with Scoreboardstat? in my server i have:
- scoreboardstats
- Title manager https://www.spigotmc.org/resources/titlemanager.1049/
- animatednames https://www.spigotmc.org/resources/animatednames.2175/
Not it doesn't have to be a other plugin ScoreboardStats uses. It's the normal player object which every plugin uses.
You can't call Player#equals in an Async environment. Which means you can't access a Player in a Map based on equality in an Async environment.
@tuanjr
does Citizen can make ScoreboardStats infinity loop?
I'm not familiar with Citizens, but if it calls the PlayerJoinEvent it could possible be.
You can't call Player#equals in an Async environment. Which means you can't access a Player in a Map based on equality in an Async environment.
I know WeakHashMap isn't thread-safe too, but the hashmap isn't even accessed async.
[20:40:36] [Paper Watchdog Thread/ERROR]: ------------------------------
[20:40:36] [Paper Watchdog Thread/ERROR]: Current Thread: Server thread
here are another crash i also ask on reddit: https://www.reddit.com/r/admincraft/comments/4u8tlm/help_me_what_plugin_make_my_server_crash/
@me4502 You should mention that.
@me4502
Isn't the thread called "Server thread" the main thread?
You're calling it when packets occur, packets are guaranteed to not occur on the main thread
So to clarify the situation. According to the stacktrace from reddit animatednames sends scoreboard packets async. The weakhashmap used in ScoreboardStats should only handle one thread. This works if everything uses the main thread. This is what you see in most plugins including mcMMO and vanilla minecraft behaviour.
If multiple threads accesses the WeakHashMap it results into a infinite loop. I'll fix it at the moment.
@me4502
It was vanilla minecraft behaviour. So I assumed to keep it. I was wrong and I'm going to fix it now. It's the first time I got this issue and this part of code exists there for ages.
Sorry for all the trouble it caused. I didn't thought about it. Thanks to @me4502 for pointing that out.
Moreover it would be better to post the complete crash report to find such spots earlier.
Sorry @me4502 for being so mean. It wasn't meant to be run async. I know player.equals isn't thread-safe, because it touches some mutable internal server things. I should really switch to uuids as keys and make the packet sending and receiving truly async.
All fine, i mistook it as packets received, which are always async. Packets that are sending, with vanilla behaviour, should be synchronised at the stage ProtocolLib touches it.
Today, i try replace ScoreboardStats by FeatherBoard, and server still crash at ~4PM, so i think server crash not cause by ScoreboardStats. here are new log (when replace Scoreboard by Featherboard), can anyone help me:
http://paste.ubuntu.com/20725210/