Type of bug

Performance issue or memory leak

/ess version output

ess version
[23:25:22 INFO]: CONSOLE issued server command: /ess version
[23:25:22 INFO]: Server version: 1.17.1-R0.1-SNAPSHOT git-Airplane-45 (MC: 1.17.1)
[23:25:22 INFO]: Brand version: Airplane
[23:25:22 INFO]: EssentialsX version: 2.19.0-dev+201-cba5063
[23:25:22 INFO]: LuckPerms version: 5.3.47
[23:25:22 INFO]: PlaceholderAPI version: 2.10.9
[23:25:22 INFO]: Vault version: 1.7.3-b131
[23:25:22 INFO]: EssentialsXProtect version: 2.19.0-dev+201-cba5063
[23:25:22 INFO]: EssentialsXSpawn version: 2.19.0-dev+201-cba5063
[23:25:22 INFO]: Citizens version: 2.0.28-SNAPSHOT (build 2346)
[23:25:22 INFO]: Economy Layer: None
[23:25:22 INFO]: Fetching version information...
[23:25:22 INFO]: You're 3 EssentialsX dev build(s) out of date!
[23:25:22 INFO]: Download it here: https://essentialsx.net/downloads.html

Server startup log

no errors present

EssentialsX config files

non applicable

Error log (if applicable)

https://spark.lucko.me/t4EkYXPJWZ

Bug description

Was noticing lag spikes in-game and decided to grab a spark to see whats going on and was surprised to see essentialsX as top offender, im not sure what is going on here but I thought I'd share this here.

https://spark.lucko.me/t4EkYXPJWZ

version
[23:27:06 INFO]: Checking version, please wait...
[23:27:06 INFO]: This server is running Airplane version git-Airplane-45 (MC: 1.17.1) (Implementing API version 1.17.1-R0.1-SNAPSHOT) (Git: 4e166c3 on HEAD)
Unknown version
Previous version: git-Airplane-44 (MC: 1.17.1)

Steps to reproduce

Run essentials on a Paper 1.17.1 server with +-100 players

Expected behaviour

No performance impact

Actual behaviour

Lag spikes & general higher mspt

Also getting crashes from EssentialsX
https://gist.github.com/cloakfox/bb605d927b48c6b55f40304194fce4b6

Please could you paste your server startup log and EssentialsX config? PlayerCommandSendEvent should not be running on a regular basis - this indicates an issue elsewhere with your setup, possibly caused by other plugins.

Please could you paste your server startup log and EssentialsX config? PlayerCommandSendEvent should not be running on a regular basis - this indicates an issue elsewhere with your setup, possibly caused by other plugins.

This is our config.yml https://gist.github.com/cloakfox/006b26fe17693f267d7230d4d5ea29af
and our startup log: https://gist.github.com/cloakfox/c2e81fed97d15d9f770e52c9a6a6a637

Is PlayerCommandSendEvent something from an EssentialsX API or something completely else, trying to understand what kind of plugin can use that and then also use it to often.

PlayerCommandSendEvent is an event that plugins can listen in 1.13+. It's fired before the server sends the list of known commands to the player for the command suggestions added in 1.13. EssentialsX listens to this so that it can hide its own commands if players don't have permissions. However, this list shouldn't be send often - the only times it is when a player logs in, they're opped or deopped, or if a plugin asks the server to update the list.

The reason this is suddenly much slower in EssentialsX 2.19.0 dev builds is due to an apparent performance regression in #4072, which removed the caching of certain config options.

Could you test this build and report back whether this makes a noticeable difference? If you could include a spark report on the new build, it would be much appreciated.

Thanks for that build, so far so good! Have been running that build for 36 minutes now and it seems like the lag spikes are completely gone now, players are also saying that they haven't noticed any lag spikes yet.

https://spark.lucko.me/TiPGOn7L3B

[15:19:34 INFO]: CONSOLE issued server command: /ess version
[15:19:34 INFO]: Server version: 1.17.1-R0.1-SNAPSHOT git-Airplane-45 (MC: 1.17.1)
[15:19:34 INFO]: Brand version: Airplane
[15:19:34 INFO]: EssentialsX version: 2.19.0-dev+219-5acedab
[15:19:34 INFO]: LuckPerms version: 5.3.47
[15:19:34 INFO]: PlaceholderAPI version: 2.10.9
[15:19:34 INFO]: Vault version: 1.7.3-b131
[15:19:34 INFO]: EssentialsXProtect version: 2.19.0-dev+219-5acedab
[15:19:34 INFO]: EssentialsXSpawn version: 2.19.0-dev+219-5acedab
[15:19:34 INFO]: Citizens version: 2.0.28-SNAPSHOT (build 2348)
[15:19:34 INFO]: Economy Layer: None
[15:19:34 INFO]: Fetching version information...
[15:19:34 INFO]: You're running an up to date experimental EssentialsX build!
[15:19:34 INFO]: Feature Branch: fix/uncached-settings-hot-paths.

getting 40 mspt with 80 players now, which is not bad considering there is 12k chunks loaded and 5k entities.

Ill report back if anything changes.

The change in my test build is now a part of standard dev builds again, so you should be able to safely update to the latest build.