Hi, can you increase speed of command Migration and Import?
My server with 200k player, using Zpermission need a night to migration.
Sometime i use export + import feature tu bulk editing permision, and import as migration need a hours to complete.
when zpermission only need some minutes to import.
Thank for great plugin and sorry for my poor English.

The speed of the migration is pretty hard to improve without splitting users up into multiple threads. I'll think about doing that though.

The /lp import command is limited in the sense that only one command can be executing at any one time. The current behaviour is to save after each command, which also slows things down.

Maybe add a system to queue up a bunch of changes before saving to storage? For example a lp init command which loads everything into memory for edits, then an lp commit command to save. Not sure how you'd deal with conflicts though, block all changes while that's happening?

Again, kinda linked to #330. (see my comments there too)

There actually are already buffers.
https://github.com/lucko/LuckPerms/blob/master/common/src/main/java/me/lucko/luckperms/common/utils/Buffer.java
https://github.com/lucko/LuckPerms/blob/master/common/src/main/java/me/lucko/luckperms/common/utils/BufferedRequest.java

However, in the case of commands, these buffers are waited on until processing is complete.

I agree, the way i/o is done needs a bit of an overhaul, I'm just not sure what the best way to do it is. I think init & commit commands would be too much of a hassle for most people - and.. I would rather try to solve the overall issue, rather than just fix things for the import command.

I see. Note that I was suggesting those commands mostly for importing, and still having the save-after-each-command system as well.

What if instead of generating commands on export, you skip the middleman and assemble the data in the format that the user is intending to import to? For example an SQL dump. Not sure how viable this would be.

Yeah, that's also possible.

To summarise the changes made above:

• Commands a no longer pushed through a single thread executor, and can execute on multiple threads concurrently.
• Unnecessary jobs, such as logging, are now skipped for the Importer process
• Import commands are inspected when the import starts, and are intelligently distributed across multiple pools. (meaning, commands which target the same user/group/track are placed in the same pool)
• The import process now runs across multiple threads, instead of sequentially.

I believe the changes outlined will significantly improve the speed of the import process, without negatively affecting consistency.

I'd be interested to hear any observations made using the latest changes.
Thanks @LapizSeller and @sweepyoface for your suggestions/help.

I've just done some testing of my own, and noticed considerable improvements. 🕺

(I exported the data used above from one of the networks I work for, so it's real data.)