Migration from Filesystem to MySQL/SQLite ends in OutOfMemory error

Question

Migration from Filesystem to MySQL/SQLite ends in OutOfMemory error

tommasobenatti opened this issue 4 months ago · 6 comments

tommasobenatti commented 4 months ago

Observed Behavior

When trying to migrate claims (like 5000 if not more), from the filesystem type of storage to MySQL or SQLite (I've tried both) the migration ends up with a OutOfMemory/Resources error and it disables itself.

I've tried giving the server 40Gb of ram with Xms and Xmx but nothing changed, and the database will not fully migrate.

Expected Behavior

Migrate the db fully (user and claim) and the plugin doesn't stop from loading and GP doesn't disable.

Reproduction steps

Have at least 5000 claims and 8000 files in PlayerData in the GP storage folders, then try migrating to MySQL/SQLite and you will get the errors shown below in the logs

Stack trace or error log

[20:40:35] [Server thread/INFO]: Preparing level "world"
[20:40:36] [Server thread/INFO]: [GriefPrevention] Enabling GriefPrevention v16.18.3
[20:40:36] [Server thread/INFO]: [GriefPrevention] Finished loading configuration.
[20:40:36] [Server thread/INFO]: [GriefPrevention] 0 total claims loaded.
[20:40:36] [Server thread/INFO]: [GriefPrevention] Customizable messages loaded.
[20:40:36] [Server thread/INFO]: [GriefPrevention] There appears to be some data on the hard drive.  Migrating those data to the database...
[20:40:36] [Server thread/INFO]: [GriefPrevention] Failed to load a claim (ID:7468) because its world isn't loaded (yet?).  If this is not expected, delete this claim.
[20:40:36] [Server thread/INFO]: [GriefPrevention] Failed to load a claim (ID:7379) because its world isn't loaded (yet?).  If this is not expected, delete this claim.
[20:40:36] [Server thread/INFO]: [GriefPrevention] Failed to load a claim (ID:7378) because its world isn't loaded (yet?).  If this is not expected, delete this claim.
[20:40:36] [Server thread/INFO]: [GriefPrevention] Failed to load a claim (ID:13149) because its world isn't loaded (yet?).  If this is not expected, delete this claim.
[20:40:36] [Server thread/INFO]: [GriefPrevention] Failed to load a claim (ID:5734) because its world isn't loaded (yet?).  If this is not expected, delete this claim.
[20:40:36] [Server thread/INFO]: [GriefPrevention] Failed to load a claim (ID:5736) because its world isn't loaded (yet?).  If this is not expected, delete this claim.
[20:40:36] [Server thread/INFO]: [GriefPrevention] 4980 total claims loaded.
[20:40:36] [Server thread/INFO]: [GriefPrevention] Customizable messages loaded.
[20:40:39] [Server thread/ERROR]: Error occurred while enabling GriefPrevention v16.18.3 (Is it up to date?)
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.lang.Thread.start0(Native Method) ~[?:?]
	at java.lang.Thread.start(Thread.java:1526) ~[?:?]
	at me.ryanhamshire.GriefPrevention.DataStore.savePlayerData(DataStore.java:1034) ~[GriefPrevention (1).jar:?]
	at me.ryanhamshire.GriefPrevention.FlatFileDataStore.migrateData(FlatFileDataStore.java:835) ~[GriefPrevention (1).jar:?]
	at me.ryanhamshire.GriefPrevention.GriefPrevention.onEnable(GriefPrevention.java:303) ~[GriefPrevention (1).jar:?]
	at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:287) ~[pufferfish-api-1.20.4-R0.1-SNAPSHOT.jar:?]
	at io.papermc.paper.plugin.manager.PaperPluginInstanceManager.enablePlugin(PaperPluginInstanceManager.java:188) ~[pufferfish-1.20.4.jar:git-Pufferfish-52]
	at io.papermc.paper.plugin.manager.PaperPluginManagerImpl.enablePlugin(PaperPluginManagerImpl.java:104) ~[pufferfish-1.20.4.jar:git-Pufferfish-52]
	at org.bukkit.plugin.SimplePluginManager.enablePlugin(SimplePluginManager.java:507) ~[pufferfish-api-1.20.4-R0.1-SNAPSHOT.jar:?]
	at org.bukkit.craftbukkit.v1_20_R3.CraftServer.enablePlugin(CraftServer.java:639) ~[pufferfish-1.20.4.jar:git-Pufferfish-52]
	at org.bukkit.craftbukkit.v1_20_R3.CraftServer.enablePlugins(CraftServer.java:550) ~[pufferfish-1.20.4.jar:git-Pufferfish-52]
	at net.minecraft.server.MinecraftServer.loadWorld0(MinecraftServer.java:672) ~[pufferfish-1.20.4.jar:git-Pufferfish-52]
	at net.minecraft.server.MinecraftServer.loadLevel(MinecraftServer.java:432) ~[pufferfish-1.20.4.jar:git-Pufferfish-52]
	at net.minecraft.server.dedicated.DedicatedServer.initServer(DedicatedServer.java:311) ~[pufferfish-1.20.4.jar:git-Pufferfish-52]
	at net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1132) ~[pufferfish-1.20.4.jar:git-Pufferfish-52]
	at net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:320) ~[pufferfish-1.20.4.jar:git-Pufferfish-52]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[20:40:39] [Server thread/INFO]: [GriefPrevention] Disabling GriefPrevention v16.18.3
[20:40:39] [Server thread/INFO]: [GriefPrevention] GriefPrevention disabled.

Server version

This server is running Pufferfish version git-Pufferfish-52 (MC: 1.20.4) (Implementing API version 1.20.4-R0.1-SNAPSHOT) (Git: b1ab664 on HEAD)

GriefPrevention version

GriefPrevention v16.18.3

Configuration

# Default values are perfect for most servers.  If you want to customize and have a question, look for the answer here first: http://dev.bukkit.org/bukkit-plugins/grief-prevention/pages/setup-and-configuration/

GriefPrevention:
  SeaLevelOverrides:
    world: -1
    world_nether: -1
    world_the_end: -1
  Claims:
    Mode:
      world_the_end: Disabled
      world: Survival
      world_nether: Survival
    PreventGlobalMonsterEggs: true
    PreventTheft: true
    ProtectCreatures: true
    PreventButtonsSwitches: true
    LockWoodenDoors: true
    LockTrapDoors: true
    LockFenceGates: true
    EnderPearlsRequireAccessTrust: true
    RaidTriggersRequireBuildTrust: true
    ProtectHorses: true
    ProtectDonkeys: true
    ProtectLlamas: true
    InitialBlocks: 20000
    Claim Blocks Accrued Per Hour:
      Default: 0
    Max Accrued Claim Blocks:
      Default: 0
    Accrued Idle Threshold: 0
    AccruedIdlePercent: 0
    AbandonReturnRatio: 1.0
    AutomaticNewPlayerClaimsRadius: 4
    AutomaticNewPlayerClaimsRadiusMinimum: 0
    ExtendIntoGroundDistance: 5
    MinimumWidth: 5
    MinimumArea: 100
    MaximumDepth: -2147483648
    InvestigationTool: STICK
    ModificationTool: GOLDEN_SHOVEL
    Expiration:
      ChestClaimDays: 7
      UnusedClaimDays: 14
      AllClaims:
        DaysInactive: -1
        ExceptWhenOwnerHasTotalClaimBlocks: 10000
        ExceptWhenOwnerHasBonusClaimBlocks: 5000
      AutomaticNatureRestoration:
        SurvivalWorlds: false
    AllowTrappedInAdminClaims: false
    MaximumNumberOfClaimsPerPlayer: 0
    CreationRequiresWorldGuardBuildPermission: true
    VillagerTradingRequiresPermission: true
    CommandsRequiringAccessTrust: /sethome
    DeliverManuals: true
    ManualDeliveryDelaySeconds: 30
    RavagersBreakBlocks: true
    FireSpreadsInClaims: false
    FireDamagesInClaims: false
    LecternReadingRequiresAccessTrust: true
  Spam:
    Enabled: false
    LoginCooldownSeconds: 60
    LoginLogoutNotificationsPerMinute: 5
    ChatSlashCommands: '[]'
    WhisperSlashCommands: '[]'
    WarningMessage: Please reduce your noise level.  Spammers will be banned.
    BanOffenders: true
    BanMessage: Banned for spam.
    AllowedIpAddresses: 1.2.3.4; 5.6.7.8
    DeathMessageCooldownSeconds: 120
    Logout Message Delay In Seconds: 0
  PvP:
    RulesEnabledInWorld:
      world: false
      world_nether: false
      world_the_end: false
    ProtectFreshSpawns: true
    PunishLogout: true
    CombatTimeoutSeconds: 15
    AllowCombatItemDrop: true
    BlockedSlashCommands: /home;/vanish;/spawn;/tpa
    ProtectPlayersInLandClaims:
      PlayerOwnedClaims: true
      AdministrativeClaims: true
      AdministrativeSubdivisions: true
    AllowLavaDumpingNearOtherPlayers:
      PvPWorlds: true
      NonPvPWorlds: false
    AllowFlintAndSteelNearOtherPlayers:
      PvPWorlds: true
      NonPvPWorlds: false
    ProtectPetsOutsideLandClaims: false
  Economy:
    ClaimBlocksMaxBonus: 0
    ClaimBlocksPurchaseCost: 0.0
    ClaimBlocksSellValue: 0.0
  ProtectItemsDroppedOnDeath:
    PvPWorlds: false
    NonPvPWorlds: false
  BlockLandClaimExplosions: true
  BlockSurfaceCreeperExplosions: false
  BlockSurfaceOtherExplosions: false
  LimitSkyTrees: true
  LimitTreeGrowth: false
  PistonMovement: CLAIMS_ONLY
  PistonExplosionSound: true
  FireSpreads: false
  FireDestroys: false
  AdminsGetWhispers: true
  AdminsGetSignNotifications: true
  VisualizationAntiCheatCompatMode: false
  SmartBan: true
  Mute New Players Using Banned Words: true
  MaxPlayersPerIpAddress: 10
  SilenceBans: true
  Siege:
    Worlds: []
    BreakableBlocks:
    - SHORT_GRASS
    - GLASS_PANE
    - GREEN_WOOL
    - BLACK_WOOL
    - OAK_PLANKS
    - GRAVEL
    - YELLOW_WOOL
    - DIRT
    - GRASS_BLOCK
    - PURPLE_WOOL
    - GRAY_WOOL
    - GLASS
    - BIRCH_PLANKS
    - FERN
    - CYAN_WOOL
    - BLUE_WOOL
    - JUNGLE_PLANKS
    - ORANGE_WOOL
    - SNOW
    - DARK_OAK_PLANKS
    - LIME_WOOL
    - PINK_WOOL
    - MAGENTA_WOOL
    - SPRUCE_PLANKS
    - RED_WOOL
    - BROWN_WOOL
    - SAND
    - COBBLESTONE
    - LIGHT_BLUE_WOOL
    - LIGHT_GRAY_WOOL
    - ACACIA_PLANKS
    - DEAD_BUSH
    - WHITE_WOOL
    DoorsOpenDelayInSeconds: 300
    CooldownEndInMinutes: 60
  EndermenMoveBlocks: false
  SilverfishBreakBlocks: false
  CreaturesTrampleCrops: false
  RabbitsEatCrops: true
  HardModeZombiesBreakDoors: false
  MobProjectilesChangeBlocks: false
  Database:
    URL: jdbc:sqlite:/home/container/dbgp.db
    UserName: ''
    Password: ''
  UseBanCommand: false
  BanCommandPattern: ban %name% %reason%
  Advanced:
    fixNegativeClaimblockAmounts: true
    ClaimExpirationCheckRate: 60
    OfflinePlayer_cache_days: 90
  Abridged Logs:
    Days To Keep: 7
    Included Entry Types:
      Social Activity: true
      Suspicious Activity: true
      Administrative Activity: false
      Debug: false
      Muted Chat Messages: false
  ConfigVersion: 1

Plugin list

None, only GriefPrevention

Running without GriefPrevention

I attempted running the server without GriefPrevention installed.
The problem does not occur when GriefPrevention is removed from the server.

Running with only GriefPrevention

I attempted running only GriefPrevention on the server.
The issue still occurs when GriefPrevention is the only plugin running.

Running on a fresh, clean server installation

I attempted testing for the issue on a new server.
The issue still occurs on a new server.

Using unmodified client

I attempted testing for the issue with the vanilla client.
The issue still occurs when using the vanilla client.

We appreciate you taking the time to fill out a bug report!

I searched for similar issues before submitting this bug report.

Jikoo · Answer 1 · 2024-09-04T00:44:42.000Z

It's probably a thread limit, not memory. If you're on Linux, check your ulimit settings.

I would also highly recommend not using a database with GP - the DatabaseDatastore implementation is so poor that it basically only has drawbacks compared to flatfile.

tommasobenatti · Answer 2 · 2024-09-04T00:47:05.000Z

I need to migrate to DB, because I need the data in a SQL, not because I want to use it

nouish · Answer 3 · 2024-09-04T01:19:18.000Z

I need to migrate to DB, because I need the data in a SQL, not because I want to use it

Be advised that you will lose data. This is a known issue with GPs MySQL implementation.

Because you say you NEED the data in SQL, I suspect you intend to integrate something with it. You'll come to find it will not play nicely, because again, it does not work the way you expect.

It does not pull data when it needs it. Instead it grabs everything when the Minecraft server starts and caches it, and then saves it again when stopping. This means any changes you make will be ignored, and then overwritten.

Good luck.

tommasobenatti · Answer 4 · 2024-09-04T02:27:43.000Z

I need the data in SQL to import onto another claim plugin that supports migration from GriefPrevention (only from SQL)

The plugin is HuskClaims.

I do not plan to use GriefPrevention on SQL Database

RoboMWM · Answer 5 · 2024-09-04T07:18:59.000Z

Migrating to SQL is discouraged, and GP's database is slated to be refactored later on in the roadmap, so there is no incentive to improve the converter. While anyone is free to look into this and potentially PR a fix, it's definitely not a priority nor on the roadmap.

Also interesting plugin. Definitely much inspired c:

RoboMWM · Answer 6 · 2024-09-04T07:19:56.000Z

Also I'm assuming you didn't intend to close the issue, but this is the rare time where the "close as not planned" state is kind of useful.

Share to