Essential userdata files
Arkobat opened this issue · 38 comments
EssentialsX version (/essentials):
Essentials 2.0.1-bSNAPSHOT
Server software (/version):
Paper Spigot 1.12.2
Essentials likes to make a bunch of userdata files.
On a previus server, we had about 600-700k userdata files after 3 month, with only 16.519 unique users joined.
On another server, we are currently at 47.108 userdata files, but only 5.460 unique users has joined the server.
Hastebin of some from the console
https://hastebin.com/nubofuduyo.vbs
We don't support offline mode servers. That said, it's most likely caused by a plugin that creates lots of NPC accounts.
@md678685
Only NPC plugin we have is citizens.
"We don't support offline mode servers."
Isn't almost every server offline serveres, because of bungeecord?
Regardless, the issue at hand is either caused by NPC accounts or attempted offline mode connections. If you are using EssentialsEco for your economy, a new "user" will be created for NPC accounts like town/faction balances. If you're actually using a server proxy, make sure you have firewall rules in place to prevent external connections except from through your proxy.
agreed - if you're running with a Waterfall or BungeeCord Server, all the Spigot or Sponge Forge Servers are running in offline mode.
Closing due to inactivity. If this is still a problem, feel free to reopen the issue.
Due to inactivity.
As I read it, it’s a problem that won’t be fixed?
External connections is prevented, but we due have a small amount of NPC’s
There had been no response between my reply in October and today.
If you have logs from when the excess files were created, could you see what is output in the logs when they are created? You should be able to search through logs for the string Creating empty config.
In addition, could you post the output from /plugins rather than just a screenshot?
New /pl list - Both text and picture, didn't know how you exactly would like it :)
https://hastebin.com/ficabapiva.cs
Checking latest.log.
The server has run from 03:30 to 17:24
There are 19453 lines in the log
About the first 13.000 is startup (We have FAWE that deletes a lot of its data when starting, which it informs us about)
The server is in Danish, so "Creating empty config" is translated to "Opretter tom konfig", which appears 1688 times in this timeline
Here are some lines after the server had started, and before many players began to log in.
https://hastebin.com/ajusokabin.md
If you need/want the full log, i can send it, but as you can see here, it says "Opretter tom konfig"(Creating empty config) a lot of times.
We are currently at 290k userdata files, but only 13,7k unique players
Could you see if any of these spam userdata files correspond to entries in your usermap.csv?
Checked 10 UUID's - No one was in usermap.csv.
The file usermap.csv has only 13785 lines, which is the same as unique players
Using this test build, could you run /ess debug, copy and post the console output when a user is created then run the command again to disable the extra logging?
Alright, here is some data.
We have 4 NPC's total on the server. 2 of them is for leaderheads, where the player "lynet96" is number 1 in both categories. (Screenshot of NPC list here: https://gyazo.com/2e031d2bad7bf9182e7f59424a07668f)
Data: https://hastebin.com/ohokaselen.vbs
I looked at it and saw it only seems to be NPC bound to leaderheads, which updates sometimes.
So I tried to spam /leaderheads update all, which forces update, and this created a lot of files
Link to that here: https://hastebin.com/ogigovivig.sql
It looks very simple the code, but I'm quite new to java, so I don't really understand it :)
But here is some debug, let me know if you need more :)
https://hastebin.com/ubujojugof.vbs
Could you give a sample of what the spammy userdata files contain?
Warning: essay ahead
After some digging around, it seems that when LeaderHeads updates its NPCs, it presumably asks Citizens to despawn the old NPC and spawn in a new one. Citizens gives each new NPC its own UUID, but this isn't based on real player UUIDs to avoid clashing with a real player or an existing NPC. Once Citizens has created the the new NPC, it spawns it in and immediately teleports the NPC's entity to the right location, causing the server to fire a PlayerTeleportEvent.
When a PlayerTeleportEvent is fired and teleport-invulnerability and/or register-back-in-listener is enabled in EssentialsX, it will try to do whichever of those options is enabled (making the user temporarily invincible and/or setting its /back location). However, to do this we need to find the right EssentialsX User, and we do this by using the Player from the teleport event to look up the right user.
When looking up a user from a Player, if a user doesn't exist for that player or UUID then we create a new EssX user and create the corresponding userdata file based on the user's UUID...
...except Citizens gives each new NPC a new UUID, so when the NPC gets deleted and recreated, the old NPC's userdata file no longer needs to exist and a new one gets created.
It's hard to say what the fix is here. Citizens has a good reason to give every single NPC a different UUID, otherwise when other plugins try to interact with them, they could mix up NPCs with players and each other. EssentialsX is only doing what it's configured to, which is to respond to players (and by extension Citizens NPCs) teleporting.
I'm not sure what causes LeaderHeads to destroy and recreate NPCs, or even if that's what it's doing at all, but if it is then it doesn't seem to take into consideration that other plugins can interact with NPCs - creating thousands of temporary NPCs can have a permanent footprint in the case of EssentialsX as well as other plugins.
I'm not sure whether EssentialsX has any mechanism for recognising Citizens NPCs as NPC users, but there's not a lot we can do other than checking whether NPCs need to store user data, which doesn't seem to be particularly easy.
TL;DR LeaderHeads might be making more NPCs than you realise, Citizens gives them all different UUIDs, EssentialsX just rolls with the excess users.
TL;DR TL;DR NPCs are people too(?)!
/essay
I actually have a zip folder on my desktop with many examples.
There are 3 folders. The first (fake) is all "fake" userdata files, where I have taken some example from different places in the folder.
The middle (Border) is where it splits from fake to real, 93 byte and under is fake, 188 byte and over is real.
The last (real) are random real player folders, but I guess you have those :)
@Arkobat What's usually inside the NPC userdata files? Could you paste an example?
Ops... Forgot the link.......... Sry.
https://ufile.io/vuvi9
If you just want a random one and not a bunch, here is a hastebin of a random
https://hastebin.com/rahopefele.css
Apologies, I forgot to actually make the test build output the stacktrace as intended - could you repeat what I said previously with this updated build?
It does look like LeaderHeads is responsible and is trying to pull EssentialsX data, but there's no way to verify that as I don't have access to the plugin or its source code. If LeaderHeads is in the stack trace, it may help identify what the issue is.
Alternatively, something could be trying to get the IUser of a Citizens NPC (that may or may not coincidentally happen to be created by LeaderHeads) causing this, and something odd happening regarding UUIDS and leading to this spam.
I've put together a simple plugin that attempts to delete the userdata file associated with an NPC when it is removed from the world. Unfortunately, this won't remove pre-existing NPC userdata files, but it might help prevent the excess creation of userdata files.
If possible, could you try and delete all the existing NPC userdata files, install CTEH and restart your server, and see whether the userdata files are still left behind when LeaderHeads updates its NPCs?
Sry for the long respond time.
I've have just installed it.
I'm currently writing a plugin to delete our old userdata files, we can't access the folder.
I will update this answer with our result tomorrow!
Edit 1
Hmmm... Had 362.268 userdata files (Total shown when using /baltop)
Spammed /leaderheads update all a few times and waited 10 minutes, now i have 362.305 files.
It seems some userdata files have been generated and not deleted.
I will check the number of files when I got home from work tomorrow (about 16 hours from now).
Debug from tonight: https://hastebin.com/pifuxogiti.md
Are you on a server that you have SSH access to?
From the userdata directory, you can run dir from Command Prompt, or ls > less from a Linux shell to list every file, and you could try typing part of the name then pressing tab to see if it autocompletes.
Otherwise, try manually typing the path to your userdata folder, followed by /<UUID>.yml to see if FileZilla can download it.
Can’t tell.
When trying to open the userdata folder, FileZilla just stops working.
Is there a command I can check with?
I can't access the files CTEH says it has deleted
Command: CWD /plugins/Essentials/userdata/5c943868-f8de-4002-88b9-3eb03b631784.yml
Answer: 550 No such file or directory.
But... Today the server had generated about 1300 more files than yesterday, so something is still "broken".
I've have finished my "hack" solution, where I can delete the invalid userdata files everytime the server starts, but I don't see this as a good solution. I find CTEH much better.
This also means, now I can access the userdata folder after I've run it the first time.
If interested, the source code can be found here, but you properly won't find it very "impressive" :)
Just out of interest, could you look for one of the spammy extra files and see if they are the same as the old extra userdata files?
Hmmmmmm...
It seems that there isn't generating any new files ATM.
I've spammed /leaderheads update all a few times, and the console is creating files, but when I update FileZilla the amount of files hasn't changed
Right now I have 15.487 files.
I'll check in tomorrow, and see if the amount has changed drastically.
If not, I don't know why I saw more files the other day, maybe I was wrong...
But for now, it seems that your plugin is working 👍
Good to hear - let us know if the problem resurfaces, and I can try and take a look. (It's possible that the number changed because FileZilla wasn't able to index every single file - not sure though.)
@ZeuosEtherus
I first user my own plugin to delete all fake files ( https://www.spigotmc.org/resources/essential-userdata-files-cleaner.53608/ )
After that, I installed CTEH, went thoug the config, and it worked.
Can’t remember if I changed anything in his config.
Hi, I'm running into the exact same issue. This may be a noob question, but after installing the fix plugin I can't seem to find the config.yml anywhere, and it's still endlessly generating the new userdata files!
@ZeuosEtherus Could you open a new issue, following the template and including a list of plugins?
I had the same problem
I created the folder myself, opened the jar with WinRar, and manually copied the config to the folder.
The name of the folder should be the same as the plugin (found in plugin.yml)
@Arkobat Ah ok, I'll try that. Thanks!