Netty memory leak clientside when on dedicated server
Barteks2x opened this issue ยท 3 comments
To reproduce: join a dedicated server and keep loading and unloading a lot of chunks (fly around very fast in already generated area). At some point you will be kicked from the server and will see this error in log:
[22:00:21] [Netty Epoll Client IO #1/ERROR] [FML]: NetworkDispatcher exception
io.netty.handler.codec.DecoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 1048496 byte(s) of direct memory (used: 1060346534, max: 1060372480)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[ByteToMessageDecoder.class:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[ByteToMessageDecoder.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [ByteToMessageDecoder.class:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) [ByteToMessageDecoder.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [ByteToMessageDecoder.class:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [ByteToMessageDecoder.class:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [ByteToMessageDecoder.class:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [ByteToMessageDecoder.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [MessageToMessageDecoder.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [IdleStateHandler.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [DefaultChannelPipeline$HeadContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [AbstractChannelHandlerContext.class:4.1.9.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [DefaultChannelPipeline.class:4.1.9.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017) [AbstractEpollStreamChannel$EpollStreamUnsafe.class:4.1.9.Final]
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:335) [AbstractEpollChannel$AbstractEpollUnsafe$1.class:4.1.9.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [AbstractEventExecutor.class:4.1.9.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) [SingleThreadEventExecutor.class:4.1.9.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) [EpollEventLoop.class:4.1.9.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [SingleThreadEventExecutor$5.class:4.1.9.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 1048496 byte(s) of direct memory (used: 1060346534, max: 1060372480)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585) ~[PlatformDependent.class:4.1.9.Final]
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539) ~[PlatformDependent.class:4.1.9.Final]
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[UnpooledUnsafeNoCleanerDirectByteBuf.class:4.1.9.Final]
at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169) ~[UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.class:4.1.9.Final]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68) ~[UnpooledUnsafeDirectByteBuf.class:4.1.9.Final]
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[UnpooledUnsafeNoCleanerDirectByteBuf.class:4.1.9.Final]
at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledByteBufAllocator.java:164) ~[UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.class:4.1.9.Final]
at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73) ~[UnpooledByteBufAllocator.class:4.1.9.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181) ~[AbstractByteBufAllocator.class:4.1.9.Final]
at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117) ~[AbstractByteBufAllocator.class:4.1.9.Final]
at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) ~[AbstractByteBuf.class:4.1.9.Final]
at net.minecraft.network.PacketBuffer.readBytes(PacketBuffer.java:959) ~[gy.class:?]
at net.minecraft.network.play.server.SPacketCustomPayload.func_148837_a(SPacketCustomPayload.java:38) ~[iw.class:?]
at net.minecraft.network.NettyPacketDecoder.decode(SourceFile:40) ~[gz.class:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[ByteToMessageDecoder.class:4.1.9.Final]
... 35 more
As I understand a several packets handled in parallel threads. Before one packet is deallocate own byte buffer another one is allocate his buffer and then they together hit a memory limit. Correct?
No. They are handled in ONE thread parallel to the main client thread, and all that thread does is schedule the handling to be done in main thread. They are handled fast enough. But at least some of the direct buffer memory (off-heap) still remains used. At some point it reaches the default 1GB limit, and you get kicked out of the server. Re-joining the server fails with the same error (because the memory is still used), but singleplayer keeps working. Joining the server after joining and quitting singleplayer still fails.
Debugging the issue is also made harder by the fact that LWJGL also uses direct buffers, because this is the only sane way to pass arrays into native code.