Ban Management

Ban Management

193k Downloads

Weird startup issue after updating to latest

tyler2k15 opened this issue ยท 49 comments

commented

[09:16:17] [Server thread/INFO] [me.confuser.banmanager.BanManager/]: [BanManager] Enabling BanManager v5.7.1
[09:16:17] [Server thread/ERROR] [Minecraft/]: Error occurred while enabling BanManager v5.7.1 (Is it up to date?)
java.lang.NullPointerException
at me.confuser.banmanager.configs.HooksConfig.(HooksConfig.java:38) ~[?:?]
at me.confuser.banmanager.configs.DefaultConfig.afterLoad(DefaultConfig.java:87) ~[?:?]
at me.confuser.banmanager.bukkitutil.configs.Config.load(Config.java:98) ~[?:?]
at me.confuser.banmanager.BanManager.setupConfigs(BanManager.java:322) ~[?:?]
at me.confuser.banmanager.BanManager.onEnable(BanManager.java:100) ~[?:?]
at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:316) ~[JavaPlugin.class:1.7.10-1448.133]
at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:368) [JavaPluginLoader.class:1.7.10-1448.133]

commented

I am aware the MySQL and SSH are different daemons. IF you aren't aware, however, you can use SSH to login to the MySQL server using command 'mysql -u [user] -p'

commented

After reverting back to version 5.3.0 which is what we were running prior, connections are now ~100-150 versus where the updated version it was maxing out connections even with increasing max connections to 3000

commented

Okey then that was just a misapprehension.

commented

I have increased my max connections to 500, but I am getting errors from servers saying there are too many connections. before updating our servers to the latest we had no issues. Now it is using all connections and i cannot even connect to it via local host ssh without max connections error

Had to revert back to previous version to even be able to use my sql

commented

MySQL and SSH are complete different daemons. Must be something else.

commented

The first issue would only occur if hooks was removed from your config.

As for the max connections, the plugin shouldn't be opening more than defined as the max in the config.

Is 5.3 the earliest version you found that didn't have this problem?

commented

I have not tested any versions between 5.3 and 5.7.1.

I can try to do that during non-peak times and see if I can come up with something and let you know.

commented

@tyler2k15 if you can narrow down the version, I can then narrow down what changes were made between that version and v5.7 to find the cause, and hopefully offer a fix/solution.

commented

@tyler2k15 I've reverted a change that was made as part of a possible fix for a different issue. This may be causing this problem.

http://ci.frostcast.net/ Build 208+

commented

To figure out which version is causing the issue I have downloaded version 1.6, 1.6.1, and 1.7.0 to test on a low-populated server.

Output of SHOW PROCESSLIST; on 1.6.0 121 rows in set (0.00 sec)
Output of SHOW PROCESSLIST; on 1.6.1 121 rows in set (0.00 sec)
Output of SHOW PROCESSLIST; on 1.7.0 119 rows in set (0.00 sec) {did notice this: 10.11 07:22:23 [Server] ERROR The time on your server and MySQL database are out by 109 seconds, this may cause syncing issues. ] not sure if that popped up other times]}
Output of SHOW PROCESSLIST; on 1.7.1 118 rows in set (0.00 sec)
Output of SHOW PROCESSLIST;Build 208
Remains the same as above. I am unsure how to reproduce unless i install it on each server and test

When testing 1.7.0 and 1.7.1 the Output of SHOW PROCESSLIST; remained the same, but after loading some of the plugins the server to seemed to freeze. This might be due to i am only testing the plugin on one server? Or maybe since I am not doing anything to call the connections it is not reproducing.

^^Will update post once I have finished tests

commented

@confuser Since my tests seemed to do no good running them on a server with low traffic, I'll test the build 208 on the entire network and inform you of the results

Other than the one option changed back, Build 208 is version 1.7.1 ?

commented

After putting all servers to build 208 it seemed fine at first
Then the connections start increasing..

Went from 118 to 300 a few minutes later, Now to 376. Everytime I run the show process list in MySQL it shows higher number

commented

What is your max connections set to in config.yml and how many servers are you running?

commented

Each server is creating unlimited number of connections:
http://prntscr.com/915nvt
Max connections: 10

12 servers total in multi craft.
7 servers + 1 Hub + 1 Network DB = 9 Databases which should only get to !100 connections.

Is now over 500 connections

commented

I found an interesting thing..

Our Modsauce 2 server timed everyone out while the SQL connections were up to 1000. I manually restarted it due to this, and now after if went down to 90 connections and now that server is up and running again it is slowly going back up.

Max connections of this server, and all servers, is set to 10 in all three locations.

Config: https://gist.github.com/tyler2k15/3ed1fb9f33c825c15695

I am now reverting This individual server back to 5.3.0, and connections are steady at 96

commented

Which database is having the connection problems, local or external? I notice from the config, you're using both.

commented

it is the ban manager_Modsauce database for the Modsauce server only that seems to open endless connections

commented

I'm afraid I'm not familiar with Modsauce 2. Is it a modded server, running under something like Cauldron?

To confirm, the other servers are running craftbukkit/spigot on v5.7 and do not have a connection problem, only the Modsauce server?

commented

All servers are modded, running cauldron. The only server not running modded is the hub. All servers are connected to the "Network" database. Umm, All but two servers are on the same box.

Mod sauce 2 is on it's own box but all servers connect to the main box's MySQL Database for Banamanager.

commented

Well now that the cause has been narrowed down somewhat, are there any MySQL based plugins you are running on Modsauce but not the other servers? Anything that interacts with BM, either via the API, or reads banmanager data from the database itself?

commented

There is one plugin. BetterEnderChest

I disabled it and tested the build 208, but still has the issue

commented

How strange. Are they all running the same build of cauldron? How are you updating BanManager, a full restart or reloading via a plugin manager?

commented

Reloading by restarting server

commented

They are not all running the same exact version. But there is at least one server running the same as Modsauce without the issue :/

commented

Cauldron versions below:

1492.149 - Agrarian Skies 2
1492.150 - Modsauce2, CrundeeCraft
1448.133 - Hubris, RR3: Project Flux, Direwolf and Infinity

commented

I haven't yet, no. I can't seem to see anything that would cause this issue to occur.

Are you running the same Java version on all servers? Is there anything config/system wise that is perhaps different?

commented

Found anything out as to why it would be isolated to this server?

commented

The server running Modsauce is located in france, meanwhile the other 2 physical servers are located in a datacenter in Canada.

Java version is 1.8.0_66-b17 on all 3 physical servers. I really can't think what is different that would cause such a database issue.

Running same version of SQL (MariaDB) - 10.0.22-MariaDB-0+deb8u1 - (Debian)

All Running Debian 8.2

Kernel Version: Linux 3.14.32-xxxx-grs-ipv6-64 on x86_64 (Same on all)

All Available package updates installed

commented

I'm seeing BanManager not obey the max connections setting. One of my servers has 79 banmanager connections opened. I'm not sure what the conditions for this are, but it's not all the time.

Will try with cf13d15

Additionally, is there a way to make BanManager fail nicely if it cannot establish connections during execution for whatever reason?

commented

Update:

It looks like it created all of those connections when our host rebooted a router, causing it to lose connection to the DB. Multiple servers spawned up to 300 connections, ultimately hitting the DB max-connections limit. Is there a potential issue with it not closing attempted connections?

commented

Perhaps. The connection handling is however lower level handled by a 3rd party library, ORMLite. Unfortunately the developer isn't overly active. This isn't the only problem I've encountered with it thus far.

commented

Try the following:

  1. Break the connection between BanManager and the DB during runtime
  2. Join a lot / force connections
  3. Check the number of connections
commented

Thanks @Iceee I'll try to reproduce it this weekend.

commented

I have the same problem with too many connections. The problem appeared out of nowhere. Running latest BM and Spigot versions.

http://ci.frostcast.net seems to be down or under maintenance, so I am not able to download the newest build.

commented

What about attempting to join very quickly?

commented

@Iceee I couldn't reproduce this locally I'm afraid. Started a server, stopped the MySQL service spam ran /bminfo around 30 - 40 times within quick succession, started MySQL. Number of connections was only ever 10, the maximum set in the default config.

commented

That was a little more difficult to simulate locally, as the connections were waiting for time outs, it slowed down the rate I could join with a single client.

commented

Perhaps the fastest solution to this is to pull in a library specifically built to handle connection pooling, rather than use ORMLite's inbuilt pool.

commented

Build 21+ running BoneCP, a different connection pool

http://ci.frostcast.net/job/BanManager/

Feedback appreciated

commented

I have installed build #24 on my server. monitoring now for max connections issue.

commented

So far its been working well. Trying build #26 now with hikaricp

commented

So far seems good. It sometimes takes a few hours to start spamming connections. I will report back later today.

commented

HikariCP seems to be more popular among the hipster database development crowd. Not particularly difficult to migrate to that from BoneCP.

The BoneCP author says that as well: https://github.com/wwadge/bonecp

It beats older connection pools such as C3P0 and DBCP but should now be considered deprecated in favour of HikariCP

commented

Wasn't aware of that, thanks @vemacs, switched.

commented

Thanks for the feedback @Posse1 ๐Ÿ‘

commented

@Posse1 any further issues?

commented

No further issues, everything is running perfectly fine with hikaricp. thanks for the quick fix.

commented

Awesome, I'll release v5.8.0 officially shortly. Thanks to everyone helping with testing.