Description

In a multi server environment with 100+ servers after assigning a permission and / or group to a user this user then looses the permission or group randomly in the future. This is an intermittent issue.

Reproduction steps

1. assign permission or group to user
2. wait indefinite amount of time
3. user reports loss of group

Expected behaviour

User does not lose group / permission

Environment details

Servers mixed 1.12.2 PaperSpigot / latest and 1.16.4 Paperspigot latest
Luckperms (Latest)
DataBase MySQL 5.7.27
Messaging - redis

620,000 approx records in _players table
850,000 approx records in _user_permissions table

DB Server on 10GB LAN with servers and running at low cpu usage.

Any other relevant details

Troubleshooting steps followed:

Verified in _actions table that insert occurred
There is no corresponding delete

We added timestamps to user_permissions table and at corresponding timestamp found missing rows (based on record ID)

We added database table triggers to log inserts and deletes and verified that the permission / group was indeed in the table and was inserted and then deleted by the luckperms user approximately 1 hr 40 mins later.

Possibly connected but unknown: at exactly the same time 2 temporary permission nodes were removed.

The most likely cause of this is another server in the network saving an old copy of user data on-top of the newer changes - this will override.

If you have any plugins that make changes to LP data via the API, ensure that they load a fresh copy of user data from the DB before making any changes and saving.

There isn't anything in LP that would randomly delete entries - in fact, user/group data isn't ever modified unless by commands or an API call.

I appreciate the detailed report, but I don't think there's anything I can do code-wise in LP to actually "fix" anything here. Hope the info above helps.