Type of bug

Performance issue or memory leak

/ess dump * output

https://essentialsx.net/dump.html?id=ebae8dd929a14c4bbe801d561de6def1

Error log (if applicable)

No response

Bug description

Hey so I've been working on this with JRoy but I figured it's probably best to make the issue on github with all the testing we've done. So I am having an issue where my server can't really stay up longer than 3 - 8 hours (3 hours on the latest stable release and 8 hours on the dev release JRoy gave me). The issue stemmed from having fake player files in my userdata which caused the error below:

https://gyazo.com/fe1645bc7ce60dfa9452e4df9048ac0b

I received a version that removed all those faulty files and now I have none. All the files in my userdata are fine and I deleted the listing.yml and the usermap.csv to let it all regenerate and all is good. Now in my server log (after many hours too), I see no errors from essentials so there is nothing indicating any errors with the plugin or my setup, however the memory leak issue still occurs.

Here is a link to a heap summary: https://spark.lucko.me/rL1f4x27lm

Some extra info to note:

• PAPI is installed however both the vault and essentials expansions are not installed. My expansions are: Bungee, griefPrevention, Javascript, LuckPerms, Player, PlayerList, PlayerPoints, Server, Superbvote, votingplugin, worldguard, Statistic
• Here is a startup log: https://pastebin.com/8ZdX6qVS

Steps to reproduce

To be honest I have no idea. It just occured for me after I updated to ess 1.17 support where configurate was introduced (or changed)

Expected behaviour

No memory leak

Actual behaviour

Memory leak

Can confirm I have the same, can give an hprof screen of the memory leak

I am seeing similar behavior, and also was having fake player files generated in my config. I am also experiencing my usermap.csv file purging itself between reboots.

Here is the one I can get. The problem I get is the memory goes from 5go to 20go in like 1-2 minutes and the server instant crash, so I can't get the big hprof scan when the ram is full

Can confirm I have the same, can give an hprof screen of the memory leak

Please do

Here is the one I can get. The problem I get is the memory goes from 5go to 20go in like 1-2 minutes and the server instant crash, so I can't get the big hprof scan when the ram is full

Do you use aikar's flags? Could I get a copy of your timings?

@JRoy Not sure if my timings would help but here they are: https://timings.aikar.co/?id=04c9c5ac2ca34ae0b2de8a41609553cc#timings

Here is the one I can get. The problem I get is the memory goes from 5go to 20go in like 1-2 minutes and the server instant crash, so I can't get the big hprof scan when the ram is full

Do you use aikar's flags? Could I get a copy of your timings?

Yes I use modified aikar flag (I use graalvm, so just some graalvm flags), and I can't do timings anymore, I reverted to the previous version with backup since it was killing my server.

I might have to rever to old essentials too if this can't be figured out. Been a really long time to wait to sort a really bad memory leak that numerous people are having. I'm down to help in this thread any way I can but please let me know what to provide

@JHarris12345 @HookWoods Would either of you be able to provide a full dump? You can send it in private on Discord to Chester#0634 via MOSS.

@JHarris12345 @HookWoods Would either of you be able to provide a full dump? You can send it in private on Discord to Chester#0634 via MOSS.

I will send it to chester today

How do you want me to send it? It's an 11GB file (server crashes whilst taking it so not sure if it managed to complete the entire thing). I don't seem to have or know of any places where I can upload an 11GB file @mdcfe

If it makes it easier, if you add me on discord JHarris#0482 I can give you file access to a server and you can download it direct from there?

Here are some screenshots from it though:

https://gyazo.com/486db5aa5edd83f14a2f9d0509b32b24

https://gyazo.com/a5d5fe5d0a02a6031ccde8024f830764

Here are some screenshots from it though:

https://gyazo.com/486db5aa5edd83f14a2f9d0509b32b24

https://gyazo.com/a5d5fe5d0a02a6031ccde8024f830764

Does the server still crash/have a "memory leak" if you remove your vote plugin?

With the vote plugin installed on my other servers I have no issues but I will try to give this a go. Hard on a live server though. Will disable voting temporarily.

I also have this screensot which shows the final path of the memory leak. Looks like its spread quite evenly over 4 segments
https://gyazo.com/2ddeaa7228a5c2d6a4c48d702551c9ee

@JRoy No the issue persists with votingplugin disabled. What do you want me to do with the heap dump. I've sent a few screenshots but might be easier to browse if someone downloads it

Heres another image 1 up from the image above https://gyazo.com/9a1934123e95bf98d7df706ed3c7b875

@mdcfe @JRoy Anything I can do to fix it on my end?

i can confirm the same issue with essentials 2.19.2

Do we have an update or a resolution to this?

This should be resolved in the last dev build

@JRoy

Unfortunately I hadn't experienced any memory leaks until I updated to the new build. Now I can't get them to stop. Not sure if I should "lol" or sad face.

Relevant pastebin: https://pastebin.com/KVPgZCV6

My server doesn't crash, surprisingly, but gets overloaded after a couple minutes of startup and eventually has to be stopped, or it would crash.

Edit: Sorry, didn't copy all the things. Updated pastebin url.

@JRoy

Unfortunately I hadn't experienced any memory leaks until I updated to the new build. Now I can't get them to stop. Not sure if I should "lol" or sad face.

Relevant pastebin: pastebin.com/CjCE4zF7

My server doesn't crash, surprisingly, but gets overloaded after a couple minutes of startup and eventually has to be stopped, or it would crash.

Edit: Sorry, didn't copy all the things. Updated pastebin url.

What storage device are you using?

@JRoy
Unfortunately I hadn't experienced any memory leaks until I updated to the new build. Now I can't get them to stop. Not sure if I should "lol" or sad face.
Relevant pastebin: pastebin.com/CjCE4zF7
My server doesn't crash, surprisingly, but gets overloaded after a couple minutes of startup and eventually has to be stopped, or it would crash.
Edit: Sorry, didn't copy all the things. Updated pastebin url.

What storage device are you using?

The system is running Linux (amd64) version "5.15.35-3-pve" and has 4 CPU threads available.
The CPU is described as an AMD Ryzen 9 5900X 12-Core Processor.
Here's my relevant spark.
https://spark.lucko.me/GzzyVtzY1t

I'm on a VPS. I have reached out to the server provider as well in case it is something on their end.

The system is running Linux (amd64) version "5.15.35-3-pve" and has 4 CPU threads available. The CPU is described as an AMD Ryzen 9 5900X 12-Core Processor. Here's my relevant spark. spark.lucko.me/GzzyVtzY1t

I'm on a VPS. I have reached out to the server provider as well in case it is something on their end.

Yeah I'd check your CPU steal time with your VPS provider, very unlikely it's an Essentials issue let alone a memory leak which should have been fixed in the most recent commit.