LuckPerms

LuckPerms

41.4k Downloads

GroupManager user migration super slow

voodootje0 opened this issue ยท 9 comments

commented

The GM migration progress outputs

14:26:13 [Server] INFO [LP] MIGRATION [GroupManager] > Starting.
14:26:13 [Server] INFO [LP] MIGRATION [GroupManager] > Starting global group migration.
14:26:15 [Server] INFO [LP] MIGRATION [GroupManager] > Migrated 30 groups in world world
14:27:37 [Server] INFO [LP] MIGRATION [GroupManager] > Migrated 500 users so far in world world
14:33:48 [Server] INFO [LP] MIGRATION [GroupManager] > Migrated 1000 users so far in world world
14:44:58 [Server] INFO [LP] MIGRATION [GroupManager] > Migrated 1500 users so far in world world
14:56:07 [Server] INFO [LP] MIGRATION [GroupManager] > Migrated 2000 users so far in world world

I can remember those migrations were like 100 times faster, but apparently something did change. I don't think it has anything todo with 3453f05 but that's the only commit that changes something with GM migration.

Also I would recommed to make the /lp migration (groupmanager true) command output the current state of the migration. Since now when you re-run the command it just displays nothing, and I was really confused of the migration was still working or not. (Since I haven't had any debug for 10 minutes)

commented

Which LP version?

Speed should have increased as a result of 2e13666

commented

4.1.90 and 4.1.45 I've tested, both very slow. Logs from above are of 4.1.90.

commented

I got some new logs, it totally took 6 hours to migrate 16734 users.
https://haste.jartexnetwork.com/imixahecer.md

Version 4.1.98 used this time.

commented

Not much I can do about this, beyond introducing multi threading.

I'd accept a decent PR that implements this, but it's not really a priority for me at the moment.

commented

I can remember you added multithreaded exporting a while ago, you even changed it into 128 pools. 8faa4d1 #317

Why is this completely removed? It was a great feature and I personally never had any issues with exporting/importing before.

commented

The import and export commands are still multi threaded, it hasn't been removed.

commented

Hmm, what was the change that made it so slow like this? Maybe I can use the version before that change, and use it only for exporting and after switch back to a recent one. I know for sure I did transfer several servers with 200k users and it just exported within ~30 minutes. And now for only 17k users it is 6 hour.

commented

import/export is not the same as migration.

The /lp import and /lp export commands are multi-threaded.
/lp migration is not.

commented

I meant to say migration instead of exporting, but thanks for your answer. I will look and try to find a solution to speed it up, and might push out a PR.