LuckPerms

LuckPerms

41.4k Downloads

Server hanging on startup

Hellohi3654 opened this issue ยท 11 comments

commented

Description

I have updated to the latest version (5.3.71) as I have started getting an issue where the server hangs on initial data load

image

The gets stuck here and stays like it until forcefully stopped.

This only happens about 50% of the time

I am using a MariaDB database and I am certain there aren't any problems with it as I have a website that also uses the database on the same VPS without issue.

Reproduction steps

  1. I don't really know

Environment details

Airplane version git-Airplane-95 (MC: 1.17.1)
Java 16

commented

When it does load properly, what is the ping to the database? You can find it using lp info - please send a screenshot of the lp info output, if you don't mind!

commented

When it does load properly, what is the ping to the database? You can find it using lp info - please send a screenshot of the lp info output, if you don't mind!

Screenshot_20211021-080620

commented

Sometimes the data doesn't work and gives this error
I have 4 servers connected to the database and they all do this at different times. The lp sync command is automatically run on all 4 servers at the same time would that be causing the issue?

Screenshot_20211021-081010

commented

you don't need to run lp sync if you haven't messed with the sync settings in the config - each server will automatically sync when needed. Try disabling whatever software is running those commands - does the problem stop?

commented

I have disabled lp sync running on all servers
Also found this stack trace on another server

Screenshot_20211021-081939

commented

101ms is very bad for databases and will probably make getting data very slow. Looking at how you have 38 groups and probably quite a lot of users, I'm not surprised it's just taking very long to load data from the database. In this case, not much luckperms can do. The error you show is just telling you the syncing task is taking unusually long, which makes sense since lp sync is a database heavy task, and your database is likely very slow.

commented

101ms is very bad for databases and will probably make getting data very slow. Looking at how you have 38 groups and probably quite a lot of users, I'm not surprised it's just taking very long to load data from the database. In this case, not much luckperms can do. The error you show is just telling you the syncing task is taking unusually long, which makes sense since lp sync is a database heavy task, and your database is likely very slow.

I understand my database is slow but when starting the server luckperms hangs on the server indefinitely and never loads
The server has to be force stopped and started again until it eventually works

commented

Is it possible to test this by running a database on the same machine as the minecraft server? You can easily lp export <anynamehere> to create an export file, and then import it to the local database by connecting LP to it and running lp import <samenamehere>

This would help us to understand if the problem is only due to the bad latency, and therefore measures need to be put in place to account for it, or if there is some other underlying issue that even low latency cannot fix. It is very difficult for us to reproduce this, having wildly different (and usually faster) connections and setups, so if you can manage to try it, it would be invaluable in diagnosing the problem.

commented

Is it possible to test this by running a database on the same machine as the minecraft server? You can easily lp export <anynamehere> to create an export file, and then import it to the local database by connecting LP to it and running lp import <samenamehere>

This would help us to understand if the problem is only due to the bad latency, and therefore measures need to be put in place to account for it, or if there is some other underlying issue that even low latency cannot fix. It is very difficult for us to reproduce this, having wildly different (and usually faster) connections and setups, so if you can manage to try it, it would be invaluable in diagnosing the problem.

I do not own the machine my server runs on
I access them through a pterodactyl panel
My MC servers run on a server in LA and my VPS which runs the database is somewhere else (I don't actually know where it is)

I also regularly export my permissions as a way of backing them up

commented

Is it possible for you to use a backup of your server to run it on a machine you do own, along with a database on the same machine? Again, I cannot replicate this problem so we'll need to try and diagnose it with your precise setup as efficiently as possible, if we are to discover a problem within LuckPerms!

commented

I'm going to go with database connection issue as the cause of this problem. Hopefully it has been resolved, but if not, we will need very clear reproduction steps to be able to take a look at this productively. Simply knowing that the database connection is spotty is our best clue towards this, as nothing else seems wrong, so we would need to be able to reproduce this on a testing setup that does not have such database issues in order to know it's a problem with LuckPerms and not the db connection. If at any point you are able to reproduce this on a server with a good connection to its database, or with your storage settings set to h2, please open a new issue, reference this one, and give clear reproduction steps. Thank you!

If anyone else searching through issues is experiencing this or something similar, please open a new issue with as much detail as possible, especially regarding database connections. As far as I can tell, that is the root of this problem and thus is outside of LuckPerms' control or ability to resolve, at least without much more specific details on each case.