Weird startup issue after updating to latest
tyler2k15 opened this issue ยท 49 comments
[09:16:17] [Server thread/INFO] [me.confuser.banmanager.BanManager/]: [BanManager] Enabling BanManager v5.7.1
[09:16:17] [Server thread/ERROR] [Minecraft/]: Error occurred while enabling BanManager v5.7.1 (Is it up to date?)
java.lang.NullPointerException
at me.confuser.banmanager.configs.HooksConfig.(HooksConfig.java:38) ~[?:?]
at me.confuser.banmanager.configs.DefaultConfig.afterLoad(DefaultConfig.java:87) ~[?:?]
at me.confuser.banmanager.bukkitutil.configs.Config.load(Config.java:98) ~[?:?]
at me.confuser.banmanager.BanManager.setupConfigs(BanManager.java:322) ~[?:?]
at me.confuser.banmanager.BanManager.onEnable(BanManager.java:100) ~[?:?]
at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:316) ~[JavaPlugin.class:1.7.10-1448.133]
at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:368) [JavaPluginLoader.class:1.7.10-1448.133]
I am aware the MySQL and SSH are different daemons. IF you aren't aware, however, you can use SSH to login to the MySQL server using command 'mysql -u [user] -p'
After reverting back to version 5.3.0 which is what we were running prior, connections are now ~100-150 versus where the updated version it was maxing out connections even with increasing max connections to 3000
I have increased my max connections to 500, but I am getting errors from servers saying there are too many connections. before updating our servers to the latest we had no issues. Now it is using all connections and i cannot even connect to it via local host ssh without max connections error
Had to revert back to previous version to even be able to use my sql
The first issue would only occur if hooks
was removed from your config.
As for the max connections, the plugin shouldn't be opening more than defined as the max in the config.
Is 5.3 the earliest version you found that didn't have this problem?
I have not tested any versions between 5.3 and 5.7.1.
I can try to do that during non-peak times and see if I can come up with something and let you know.
@tyler2k15 if you can narrow down the version, I can then narrow down what changes were made between that version and v5.7 to find the cause, and hopefully offer a fix/solution.
@tyler2k15 I've reverted a change that was made as part of a possible fix for a different issue. This may be causing this problem.
http://ci.frostcast.net/ Build 208+
To figure out which version is causing the issue I have downloaded version 1.6, 1.6.1, and 1.7.0 to test on a low-populated server.
Output of SHOW PROCESSLIST;
on 1.6.0 121 rows in set (0.00 sec)
Output of SHOW PROCESSLIST;
on 1.6.1 121 rows in set (0.00 sec)
Output of SHOW PROCESSLIST;
on 1.7.0 119 rows in set (0.00 sec) {did notice this: 10.11 07:22:23 [Server] ERROR The time on your server and MySQL database are out by 109 seconds, this may cause syncing issues. ] not sure if that popped up other times]}
Output of SHOW PROCESSLIST;
on 1.7.1 118 rows in set (0.00 sec)
Output of SHOW PROCESSLIST;
Build 208
Remains the same as above. I am unsure how to reproduce unless i install it on each server and test
When testing 1.7.0 and 1.7.1 the Output of SHOW PROCESSLIST;
remained the same, but after loading some of the plugins the server to seemed to freeze. This might be due to i am only testing the plugin on one server? Or maybe since I am not doing anything to call the connections it is not reproducing.
^^Will update post once I have finished tests
@confuser Since my tests seemed to do no good running them on a server with low traffic, I'll test the build 208 on the entire network and inform you of the results
Other than the one option changed back, Build 208 is version 1.7.1 ?
After putting all servers to build 208 it seemed fine at first
Then the connections start increasing..
Went from 118 to 300 a few minutes later, Now to 376. Everytime I run the show process list in MySQL it shows higher number
What is your max connections set to in config.yml
and how many servers are you running?
Each server is creating unlimited number of connections:
http://prntscr.com/915nvt
Max connections: 10
12 servers total in multi craft.
7 servers + 1 Hub + 1 Network DB = 9 Databases which should only get to !100 connections.
Is now over 500 connections
I found an interesting thing..
Our Modsauce 2 server timed everyone out while the SQL connections were up to 1000. I manually restarted it due to this, and now after if went down to 90 connections and now that server is up and running again it is slowly going back up.
Max connections of this server, and all servers, is set to 10 in all three locations.
Config: https://gist.github.com/tyler2k15/3ed1fb9f33c825c15695
I am now reverting This individual server back to 5.3.0, and connections are steady at 96
Which database is having the connection problems, local or external? I notice from the config, you're using both.
it is the ban manager_Modsauce database for the Modsauce server only that seems to open endless connections
I'm afraid I'm not familiar with Modsauce 2
. Is it a modded server, running under something like Cauldron?
To confirm, the other servers are running craftbukkit/spigot on v5.7 and do not have a connection problem, only the Modsauce server?
All servers are modded, running cauldron. The only server not running modded is the hub. All servers are connected to the "Network" database. Umm, All but two servers are on the same box.
Mod sauce 2 is on it's own box but all servers connect to the main box's MySQL Database for Banamanager.
Well now that the cause has been narrowed down somewhat, are there any MySQL based plugins you are running on Modsauce but not the other servers? Anything that interacts with BM, either via the API, or reads banmanager data from the database itself?
There is one plugin. BetterEnderChest
I disabled it and tested the build 208, but still has the issue
How strange. Are they all running the same build of cauldron? How are you updating BanManager, a full restart or reloading via a plugin manager?
They are not all running the same exact version. But there is at least one server running the same as Modsauce without the issue :/
Cauldron versions below:
1492.149 - Agrarian Skies 2
1492.150 - Modsauce2, CrundeeCraft
1448.133 - Hubris, RR3: Project Flux, Direwolf and Infinity
I haven't yet, no. I can't seem to see anything that would cause this issue to occur.
Are you running the same Java version on all servers? Is there anything config/system wise that is perhaps different?
The server running Modsauce is located in france, meanwhile the other 2 physical servers are located in a datacenter in Canada.
Java version is 1.8.0_66-b17 on all 3 physical servers. I really can't think what is different that would cause such a database issue.
Running same version of SQL (MariaDB) - 10.0.22-MariaDB-0+deb8u1 - (Debian)
All Running Debian 8.2
Kernel Version: Linux 3.14.32-xxxx-grs-ipv6-64 on x86_64 (Same on all)
All Available package updates installed
I'm seeing BanManager not obey the max connections setting. One of my servers has 79 banmanager connections opened. I'm not sure what the conditions for this are, but it's not all the time.
Will try with cf13d15
Additionally, is there a way to make BanManager fail nicely if it cannot establish connections during execution for whatever reason?
Update:
It looks like it created all of those connections when our host rebooted a router, causing it to lose connection to the DB. Multiple servers spawned up to 300 connections, ultimately hitting the DB max-connections limit. Is there a potential issue with it not closing attempted connections?
Perhaps. The connection handling is however lower level handled by a 3rd party library, ORMLite. Unfortunately the developer isn't overly active. This isn't the only problem I've encountered with it thus far.
Try the following:
- Break the connection between BanManager and the DB during runtime
- Join a lot / force connections
- Check the number of connections
Thanks @Iceee I'll try to reproduce it this weekend.
I have the same problem with too many connections. The problem appeared out of nowhere. Running latest BM and Spigot versions.
http://ci.frostcast.net seems to be down or under maintenance, so I am not able to download the newest build.
@Iceee I couldn't reproduce this locally I'm afraid. Started a server, stopped the MySQL service spam ran /bminfo around 30 - 40 times within quick succession, started MySQL. Number of connections was only ever 10, the maximum set in the default config.
That was a little more difficult to simulate locally, as the connections were waiting for time outs, it slowed down the rate I could join with a single client.
Perhaps the fastest solution to this is to pull in a library specifically built to handle connection pooling, rather than use ORMLite's inbuilt pool.
Build 21+ running BoneCP, a different connection pool
http://ci.frostcast.net/job/BanManager/
Feedback appreciated
I have installed build #24 on my server. monitoring now for max connections issue.
So far its been working well. Trying build #26 now with hikaricp
So far seems good. It sometimes takes a few hours to start spamming connections. I will report back later today.
HikariCP seems to be more popular among the hipster database development crowd. Not particularly difficult to migrate to that from BoneCP.
The BoneCP author says that as well: https://github.com/wwadge/bonecp
It beats older connection pools such as C3P0 and DBCP but should now be considered deprecated in favour of HikariCP
Wasn't aware of that, thanks @vemacs, switched.
Thanks for the feedback @Posse1 ๐
@Posse1 any further issues?
No further issues, everything is running perfectly fine with hikaricp. thanks for the quick fix.