Latest update causing severe performance issues with wildstacker/GPflags installed

Question

Latest update causing severe performance issues with wildstacker/GPflags installed

alek4032 opened this issue 9 months ago · 8 comments

alek4032 commented 9 months ago

Observed Behavior

Hey.

We're experiencing severe tps issues after installing the latest version of Griefprevention. I'll attach 2 different spark reports and they'll both show how Wildstacker and Gpflags is using a lot more server thread for me.ryanhamshire.GriefPrevention.DataStore.getClaim()

We haven't modified anything else for the plugin other than the update, and there also has not been any kind of spike in claimflags used or something like that from what we can tell.

We've never had this issue previously before we did the update and would really appreciate if there's a way we can resolve it.

Expected Behavior

All plugins going back to using the normal of server thread (which has never been anywhere near this before)

Reproduction steps

1: Install Griefprevention
2: Install Wildstacker (Build 2023.3-b34)
3: Install latest version of GPflags (5.12.10)
4: Have them fun for a while with a lot of players on interacting with the claims as they usually do.

Stack trace or error log

There is no stack trace or error log but here's 2 different spark reports showing the issue appearing:


https://spark.lucko.me/k7fkPwkde2

and another one that was made later

https://spark.lucko.me/IUf9oBevg9

Server version

[21:29:33 INFO]: This server is running Pufferfish+ version git-PufferfishPlus-"a9f972b" (MC: 1.20.1) (Implementing API version 1.20.1-R0.1-SNAPSHOT) (Git: a9f972b on HEAD)

GriefPrevention version

GriefPrevention version 16.18.2

Configuration

# Default values are perfect for most servers.  If you want to customize and have a question, look for the answer here first: http://dev.bukkit.org/bukkit-plugins/grief-prevention/pages/setup-and-configuration/

GriefPrevention:
  SeaLevelOverrides:
    world: -1
    world_nether: 256
    world_the_end: -1
    resources: -1
    newspawn: -1
  Claims:
    Mode:
      resources: Disabled
      world_the_end: Disabled
      world_nether: Survival
      newspawn: Survival
      world: Survival
    PreventGlobalMonsterEggs: true
    PreventTheft: true
    ProtectCreatures: true
    PreventButtonsSwitches: true
    LockWoodenDoors: false
    LockTrapDoors: false
    LockFenceGates: true
    EnderPearlsRequireAccessTrust: true
    RaidTriggersRequireBuildTrust: true
    ProtectHorses: true
    ProtectDonkeys: true
    ProtectLlamas: true
    InitialBlocks: 2000
    Claim Blocks Accrued Per Hour:
      Default: 75
    Max Accrued Claim Blocks:
      Default: 80000
    Accrued Idle Threshold: 0
    AccruedIdlePercent: 0
    AbandonReturnRatio: 1.0
    AutomaticNewPlayerClaimsRadius: 4
    AutomaticNewPlayerClaimsRadiusMinimum: 0
    ExtendIntoGroundDistance: 5
    MinimumWidth: 5
    MinimumArea: 100
    MaximumDepth: -64
    InvestigationTool: STICK
    ModificationTool: GOLDEN_SHOVEL
    Expiration:
      ChestClaimDays: 7
      UnusedClaimDays: 14
      AllClaims:
        DaysInactive: 60
        ExceptWhenOwnerHasTotalClaimBlocks: 10000
        ExceptWhenOwnerHasBonusClaimBlocks: 5000
      AutomaticNatureRestoration:
        SurvivalWorlds: false
    AllowTrappedInAdminClaims: false
    MaximumNumberOfClaimsPerPlayer: 0
    CreationRequiresWorldGuardBuildPermission: true
    VillagerTradingRequiresPermission: true
    CommandsRequiringAccessTrust: /sethome;/pw set;/sign edit;/sign clear;/sign paste;/sign
      set;/editsign;/editsign set;/editsign clear;/editsign copy;/editsign paste
    DeliverManuals: true
    ManualDeliveryDelaySeconds: 30
    RavagersBreakBlocks: true
    FireSpreadsInClaims: false
    FireDamagesInClaims: false
    LecternReadingRequiresAccessTrust: true
  Spam:
    Enabled: false
    LoginCooldownSeconds: 0
    LoginLogoutNotificationsPerMinute: 5
    ChatSlashCommands: /me;/global;/local
    WhisperSlashCommands: /tell;/pm;/r;/whisper;/msg
    WarningMessage: Please reduce your noise level. Check /rules for further details
    BanOffenders: false
    BanMessage: Kicked for spam. Please refer to the rules in our discord https://discord.gg/pixelblock
    AllowedIpAddresses: 1.2.3.4; 5.6.7.8
    DeathMessageCooldownSeconds: 120
    Logout Message Delay In Seconds: 0
  PvP:
    RulesEnabledInWorld:
      world: true
      world_nether: true
      world_the_end: true
      resources: true
      newspawn: true
    ProtectFreshSpawns: true
    PunishLogout: false
    CombatTimeoutSeconds: 0
    AllowCombatItemDrop: false
    BlockedSlashCommands: /home;/vanish;/spawn;/tpa
    ProtectPlayersInLandClaims:
      PlayerOwnedClaims: true
      AdministrativeClaims: true
      AdministrativeSubdivisions: true
    AllowLavaDumpingNearOtherPlayers:
      PvPWorlds: true
      NonPvPWorlds: false
    AllowFlintAndSteelNearOtherPlayers:
      PvPWorlds: true
      NonPvPWorlds: false
    ProtectPetsOutsideLandClaims: false
  Economy:
    ClaimBlocksMaxBonus: 0
    ClaimBlocksPurchaseCost: 0.0
    ClaimBlocksSellValue: 0.0
  ProtectItemsDroppedOnDeath:
    PvPWorlds: false
    NonPvPWorlds: true
  BlockLandClaimExplosions: true
  BlockSurfaceCreeperExplosions: true
  BlockSurfaceOtherExplosions: true
  LimitSkyTrees: true
  LimitTreeGrowth: false
  PistonMovement: CLAIMS_ONLY
  PistonExplosionSound: true
  FireSpreads: false
  FireDestroys: false
  AdminsGetWhispers: true
  AdminsGetSignNotifications: true
  VisualizationAntiCheatCompatMode: false
  SmartBan: true
  Mute New Players Using Banned Words: true
  MaxPlayersPerIpAddress: 3
  SilenceBans: true
  Siege:
    Worlds: []
    BreakableBlocks:
    - GRASS_BLOCK
    - DIRT
    - COBBLESTONE
    - OAK_PLANKS
    - SPRUCE_PLANKS
    - BIRCH_PLANKS
    - JUNGLE_PLANKS
    - ACACIA_PLANKS
    - DARK_OAK_PLANKS
    - SAND
    - GRAVEL
    - GLASS
    - GRASS
    - FERN
    - DEAD_BUSH
    - WHITE_WOOL
    - ORANGE_WOOL
    - MAGENTA_WOOL
    - LIGHT_BLUE_WOOL
    - YELLOW_WOOL
    - LIME_WOOL
    - PINK_WOOL
    - GRAY_WOOL
    - LIGHT_GRAY_WOOL
    - CYAN_WOOL
    - PURPLE_WOOL
    - BLUE_WOOL
    - BROWN_WOOL
    - GREEN_WOOL
    - RED_WOOL
    - BLACK_WOOL
    - SNOW
    - GLASS_PANE
    DoorsOpenDelayInSeconds: 300
    CooldownEndInMinutes: 60
  EndermenMoveBlocks: false
  SilverfishBreakBlocks: false
  CreaturesTrampleCrops: false
  RabbitsEatCrops: true
  HardModeZombiesBreakDoors: false
  MobProjectilesChangeBlocks: false
  Database:
    URL: ''
    UserName: ''
    Password: ''
  UseBanCommand: false
  BanCommandPattern: ban %name% %reason%
  Advanced:
    fixNegativeClaimblockAmounts: true
    ClaimExpirationCheckRate: 60
    OfflinePlayer_cache_days: 90
  Abridged Logs:
    Days To Keep: 7
    Included Entry Types:
      Social Activity: true
      Suspicious Activity: true
      Administrative Activity: true
      Debug: false
      Muted Chat Messages: false
  ConfigVersion: 1

Plugin list

[21:27:21 INFO]: Paper Plugins:
[21:27:21 INFO]:  - EconomyShopGUI, Minepacks
[21:27:21 INFO]: Bukkit Plugins:
[21:27:21 INFO]:  - AdvancedChests, AdvancedEnchantments, AlexManager, ArmorStandEditor, BattlePass, *BCommons, BetterRTP, BungeeGuard, BuycraftX, ClueScrolls
[21:27:21 INFO]:  CommandWhitelist, CoreProtect, CrazyEnvoys, CustomDecentHolograms, DailyRewardsPlus, DecentHolograms, DeluxeCoinflip, DiscordSRV, Duels, EliteArmor
[21:27:21 INFO]:  EntityDetection, Essentials, EssentialsChat, EssentialsSpawn, ExcellentCrates, ExecutableBlocks, ExecutableItems, FancyNpcs, FarmLimiter, GriefPrevention
[21:27:21 INFO]:  GSit, GUIPlus, Harbor, HeadDatabase, Insights, InteractiveChat, InventoryRollbackPlus, InvSeePlusPlus, LibsDisguises, LiteBans
[21:27:21 INFO]:  LuckPerms, mcMMO, MCPets, MobLimit, ModelEngine, Multiverse-Core, MythicCrucible, MythicMobs, NexEngine, PBBlockLocker
[21:27:21 INFO]:  PBGPFlagManager, PBManager, PBNotifier, PBPackLoader, PinataParty, PlaceholderAPI, Plan, PlayerVaults, PlayerWarps, PlugManX
[21:27:21 INFO]:  ProCosmetics, ProtocolLib, PyroFishingPro, PyroLib, Quests, QuickShop, SCore, Shopkeepers, SimpleClans, SimplePortals
[21:27:21 INFO]:  spark, TAB, TempFly, TogglePvp, TradeSystem, UpgradeableHoppers, Vault, ViaBackwards, ViaVersion, ViewDistanceTweaks
[21:27:21 INFO]:  VoidGen, Votifier, WildBuster, WildChests, WildStacker, WildTools, WordFilter, WorldEdit, WorldGuard, WorldGuardExtraFlags
[21:27:21 INFO]:  zAuctionHouseV3

Running without GriefPrevention

I attempted running the server without GriefPrevention installed.
The problem does not occur when GriefPrevention is removed from the server.

Running with only GriefPrevention

I attempted running only GriefPrevention on the server.
The issue still occurs when GriefPrevention is the only plugin running.

Running on a fresh, clean server installation

I attempted testing for the issue on a new server.
The issue still occurs on a new server.

Using unmodified client

I attempted testing for the issue with the vanilla client.
The issue still occurs when using the vanilla client.

We appreciate you taking the time to fill out a bug report!

I searched for similar issues before submitting this bug report.

alek4032 · Answer 1 · 2024-03-10T00:10:31.000Z

Adding on to the pile of "probably not GP" - the only changes to the DataStore since 16.18.1 are a single new logging line and message edits. https://github.com/GriefPrevention/GriefPrevention/compare/16.18.1...16.18.2?diff=unified&w=1#diff-3b102842811e8a55c46e8fa82bb6cca9de9418f9859fa3550efa762af82232b0

Honestly weirdest part to me is that those look like EntityDamageEvents. Why are those plugins fetching claims via Datastore#getClaim instead of DataStore#getClaimAt? Fetching claim by ID is wildly suboptimal.

/e: Yeah I'm willing to say that it might not be just the 20k entities that are a problem here: https://github.com/akdukaan/GPFlags/blob/a7ab8e9c27836073afac0b24b81199948fab26e1/src/main/java/me/ryanhamshire/GPFlags/flags/FlagDefinition.java#L80-L86 https://github.com/akdukaan/GPFlags/blob/master/src/main/java/me/ryanhamshire/GPFlags/FlagManager.java#L191-L228 Looks like a BigScary design decision to store flags per claim ID as strings. The subsequent modifications have compounded the effects of that because now it performs at least double (if not quadruple?) the claim-to-long-to-string-to-long-to-claim lookups.

Yeah I agree, something was changed. I found another spark report that shows even more than 20k entities, almost 100 players and GPflags version v5.12.8 only using Server thread 0.29%. That's 1 version older than the we one updated to. In the same spark report Griefprevention also only use Server thread 0.25% with the older version (v16.18.1)

alek4032 · Answer 2 · 2024-03-10T01:15:47.000Z

We downgraded GPflags to a version older and that resolved all the issues.

RoboMWM · Answer 3 · 2024-03-09T22:10:36.000Z

idk if I'm dumb or something but how does it figure that it's taking 7.48% of the tick and the breakdown doesn't equate to that? Does that means other things are contributing to it taking time? Either way, I wonder if it's cuz maybe it's waiting for some event it calls to return by some other plugin handling it or something idk.

RoboMWM · Answer 4 · 2024-03-09T22:13:01.000Z

Anyways, curious if you observe any difference if you remove wildstacker and gpflags.

RoboMWM · Answer 5 · 2024-03-09T22:40:02.000Z

So I asked some other ppl for right now, basically they're seeing what I'm seeing, which is a bazillion entities as being the more major issue. Idk if you have a report of an earlier version of GP to compare with same setup as these reports.

In my opinion, seems like it may be better to address entities properly instead of via stacker plugins.

the problem in the reports is not gp, it's the absolute batshit insane amount of entities
make 20k calls to any map get 20 times a second and everything will take long

YouHaveTrouble · Answer 6 · 2024-03-09T22:40:38.000Z

My brother in christ, you have 20k entities. Do a lookup for anything 20k times and it's gonna take long.
Issue is not the plugins, it's the batshit insane amount of entities.

Jikoo · Answer 7 · 2024-03-09T22:51:34.000Z

Adding on to the pile of "probably not GP" - the only changes to the DataStore since 16.18.1 are a single new logging line and message edits. https://github.com/GriefPrevention/GriefPrevention/compare/16.18.1...16.18.2?diff=unified&w=1#diff-3b102842811e8a55c46e8fa82bb6cca9de9418f9859fa3550efa762af82232b0

Honestly weirdest part to me is that those look like EntityDamageEvents. Why are those plugins fetching claims via Datastore#getClaim instead of DataStore#getClaimAt? Fetching claim by ID is wildly suboptimal.

/e: Yeah I'm willing to say that it might not be just the 20k entities that are a problem here:
https://github.com/akdukaan/GPFlags/blob/a7ab8e9c27836073afac0b24b81199948fab26e1/src/main/java/me/ryanhamshire/GPFlags/flags/FlagDefinition.java#L80-L86
https://github.com/akdukaan/GPFlags/blob/master/src/main/java/me/ryanhamshire/GPFlags/FlagManager.java#L191-L228
Looks like a BigScary design decision to store flags per claim ID as strings. The subsequent modifications have compounded the effects of that because now it performs at least double (if not quadruple?) the claim-to-long-to-string-to-long-to-claim lookups.

akdukaan · Answer 8 · 2024-08-11T21:43:56.000Z

@alek4032 I reworked the way GPF looks up claims in v5.13.1 to fix a lot of these performance issues it had. I think it's now more optimized than any prior version that I'm aware of. If you're still using the older version, I recommend updating to the latest GPFlags!

Share to