Option for local MySQL requests be async
ice-fly opened this issue · 12 comments
I've been experiencing server lag... I am running paper-298 (currently) and have been trying to track down why I'm seeing lag spikes.
After many timings reports & a suspicion of Dynmap, I set the write interval on the JsonFileClientUpdateComponent writeinterval
to 35
Sure enough, every 35 seconds my server chugs...
[13:13:33 INFO]: Server tick times (avg/min/max) from last 5s, 10s, 1m:
[13:13:31 INFO]: ◴ 1.7/0.3/93.9, 1.2/0.3/93.9, 0.9/0.2/94.5
[13:13:48 INFO]: ◴ 0.7/0.2/2.0, 0.7/0.2/2.3, 0.9/0.2/94.5
...
[13:14:05 INFO]: ◴ 0.7/0.3/2.1, 0.7/0.3/2.1, 0.8/0.2/93.9
[13:14:06 INFO]: ◴ 1.7/0.3/93.8, 1.2/0.3/93.8, 0.9/0.2/93.9
I suspect this issue is often not as easy to detect as most people are running MySQL on their machine. I have a tunnel setup to a web-host that runs the SQL database & dynmap standalone. Hence, my access times are longer & why I have a noticeable slowdown to my server.
Also, disconnects from jdbc make server unusable due to constant re-connect requests.
The fix to this is to move dynmap's database operations off the main thread & make them asynchronous.
Reading between the lines on #2775 could it be that when localhost is the target address, the plugin expects fast access times? And thus doesn't bother with async?
I think getConnection() needs async:
An error log... git-Paper-318 (MC: 1.15.2) :
Error
[02:05:29 ERROR]: Current Thread: Server thread [02:05:29 ERROR]: PID: 25 | Suspended: false | Native: false | State: WAITING [02:05:29 ERROR]: Stack: [02:05:29 ERROR]: [email protected]/java.lang.Object.wait(Native Method) [02:05:29 ERROR]: [email protected]/java.lang.Object.wait(Unknown Source) [02:05:29 ERROR]: org.dynmap.storage.mysql.MySQLMapStorage.getConnection(MySQLMapStorage.java:526) [02:05:29 ERROR]: org.dynmap.storage.mysql.MySQLMapStorage.getStandaloneFile(MySQLMapStorage.java:989) [02:05:29 ERROR]: org.dynmap.JsonFileClientUpdateComponent.handleWebChat(JsonFileClientUpdateComponent.java:373) [02:05:29 ERROR]: org.dynmap.JsonFileClientUpdateComponent$1.run(JsonFileClientUpdateComponent.java:152) [02:05:29 ERROR]: app//org.bukkit.craftbukkit.v1_15_R1.scheduler.CraftTask.run(CraftTask.java:84) [02:05:29 ERROR]: app//org.bukkit.craftbukkit.v1_15_R1.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:461) [02:05:29 ERROR]: app//net.minecraft.server.v1_15_R1.MinecraftServer.b(MinecraftServer.java:1256) [02:05:29 ERROR]: app//net.minecraft.server.v1_15_R1.DedicatedServer.b(DedicatedServer.java:430) [02:05:29 ERROR]: app//net.minecraft.server.v1_15_R1.MinecraftServer.a(MinecraftServer.java:1173) [02:05:29 ERROR]: app//net.minecraft.server.v1_15_R1.MinecraftServer.run(MinecraftServer.java:962) [02:05:29 ERROR]: [email protected]/java.lang.Thread.run(Unknown Source)
I'd rather dynmap render slower than chew up my main thread and cause tps lag
OK - just to be clear - your scenario is using MySQL as a storage type, and running with an external server. I'm doing some problem determination now, and I'm seeing calls to the storage from the server thread when words are activated, but I haven't found anything else yet. Are you using any other plugins that might be calling the Dynmap APIs for, say, markers or the like? Just trying to find the cases to be pinned down here - straight up Spigot + Dynmap with external web doesn't seem to be making storage calls from the server thread, except on what would be relatively rare cases.
Yes, I have a particularly egregious case: The sql server has ping times of ~100ms. (Shoestring mc server budget) I'm using a webhost who provides me a shared webserver to host server website, sql database, & dynmap standalone files.
I'm doing a full render, but cpu never goes above 50% utilization
Dynmap version: core=3.0-RC2-318, plugin=3.0-RC2-318 (just realized I'm out of date, updating now)
Plugins (22)
This seemed to happen before Dynmap-GriefPrevention, snowdrift, & snowier world were installed. Not that it isn't them... (just less likely) AsyncWorldEdit, AutoRestart*, ChopTree2, ClaimslistClassifier, DiscordSRV, dynmap*, Dynmap-GriefPrevention*, Essentials, EssentialsAntiBuild, EssentialsChat, EssentialsProtect, EssentialsSpawn, GriefPrevention, LuckPerms, MineableSpawners, OreAnnouncer, ProtocolLib, SnowDrift*, SnowierSnow, Vault, WorldEdit, WorldGuardConfig
Dynmap: ``` # All ... hires user 'userid' password 'password' and table prefix 'prefix')
type: mysql
hostname: localhost
port: 24306
database: (Obfuscated)
userid: (Obfuscated)
password: (Obfuscated)
prefix: ""
flags: "?allowReconnect=true"
components:
- class: org.dynmap.ClientConfigurationComponent
...
- class: org.dynmap.JsonFileClientUpdateComponent
writeinterval: 6
sendhealth: false
sendposition: true
allowwebchat: true
webchat-interval: 12
hidewebchatip: true
includehiddenplayers: false
use-name-colors: true
use-player-login-ip: true
require-player-login-ip: true
block-banned-player-chat: true
hideifshadow: 8
hideifundercover: 8
hideifsneaking: true
# Require login for web-to-server chat (requires login-enabled: true)
webchat-requires-login: true
# If set to true, users must have dynmap.webchat permission in order to chat
webchat-permissions: true
# Limit length of single chat messages
chatlengthlimit: 250
hide-if-invisiblity-potion: true
hidenames: false
- class: org.dynmap.SimpleWebChatComponent
allowchat: true
# If true, web UI users can supply name for chat using 'playername' URL parameter. 'trustclientname' must also be set true.
allowurlname: false
# Note: this component is needed for the dmarker commands, and for the Marker API to be available to other plugins
- class: org.dynmap.MarkersComponent
type: markers
showlabel: true
enablesigns: true
# Default marker set for sign markers
default-sign-set: playerMarkers
# (optional) add spawn point markers to standard marker layer
showspawn: true
spawnicon: world
spawnlabel: "Spawn"
# (optional) layer for showing offline player's positions (for 'maxofflinetime' minutes after logoff)
showofflineplayers: false
offlinelabel: "Offline"
offlineicon: offlineuser
offlinehidebydefault: true
offlineminzoom: 2
maxofflinetime: 120
# (optional) layer for showing player's spawn beds
showspawnbeds: false
spawnbedlabel: "Spawn Beds"
spawnbedicon: bed
spawnbedhidebydefault: true
spawnbedminzoom: 0
spawnbedformat: "%name%'s bed"
# (optional) show world border (vanilla 1.8+)
showworldborder: true
- class: org.dynmap.ClientComponent
type: chat
allowurlname: false
- class: org.dynmap.ClientComponent
type: chatballoon
focuschatballoons: false
- class: org.dynmap.ClientComponent
type: chatbox
showplayerfaces: true
messagettl: 8
# Optional: set number of lines in scrollable message history: if set, messagettl is not used to age out messages
scrollback: 80
# Optional: set maximum number of lines visible for chatbox
visiblelines: 8
# Optional: send push button
sendbutton: true
- class: org.dynmap.ClientComponent
type: playermarkers
showplayerfaces: true
showplayerhealth: false
# If true, show player body too (only valid if showplayerfaces=true
showplayerbody: false
# Option to make player faces small - don't use with showplayerhealth
smallplayerfaces: true
# Optional - make player faces layer hidden by default
hidebydefault: false
# Optional - ordering priority in layer menu (low goes before high - default is 0)
layerprio: 0
# Optional - label for player marker layer (default is 'Players')
label: "Players"
#- class: org.dynmap.ClientComponent
# type: digitalclock
- class: org.dynmap.ClientComponent
type: link
- class: org.dynmap.ClientComponent
type: timeofdayclock
showdigitalclock: true
showweather: true
# Mouse pointer world coordinate display
- class: org.dynmap.ClientComponent
type: coord
label: "Location"
hidey: false
show-mcr: false
# Note: more than one logo component can be defined
#- class: org.dynmap.ClientComponent
# type: logo
# text: "Dynmap"
# #logourl: "images/block_surface.png"
# linkurl: "http://forums.bukkit.org/threads/dynmap.489/"
# # Valid positions: top-left, top-right, bottom-left, bottom-right
# position: bottom-right
- class: org.dynmap.ClientComponent
type: inactive
timeout: 1800 # in seconds (1800 seconds = 30 minutes)
redirecturl: inactive.html
showmessage: 'You were inactive for too long.'
#- class: org.dynmap.TestComponent
# stuff: "This is some configuration-value"
# Treat hiddenplayers.txt as a whitelist for players to be shown on the map? (Default false)
display-whitelist: false
# How often a tile gets rendered (in seconds).
renderinterval: 0.64
# How many tiles on update queue before accelerate render interval
renderacceleratethreshold: 64
# How often to render tiles when backlog is above renderacceleratethreshold
renderaccelerateinterval: 0.32
# How many update tiles to work on at once (if not defined, default is 1/2 the number of cores)
tiles-rendered-at-once: 2
# If true, use normal priority threads for rendering (versus low priority) - this can keep rendering
# from starving on busy Windows boxes (Linux JVMs pretty much ignore thread priority), but may result
# in more competition for CPU resources with other processes
usenormalthreadpriority: false
# Save and restore pending tile renders - prevents their loss on server shutdown or /reload
saverestorepending: true
# Save period for pending jobs (in seconds): periodic saving for crash recovery of jobs
save-pending-period: 960
# Zoom-out tile update period - how often to scan for and process tile updates into zoom-out tiles (in seconds)
zoomoutperiod: 8
# Control whether zoom out tiles are validated on startup (can be needed if zoomout processing is interrupted, but can be expensive on large maps)
initial-zoomout-validate: true
# Default delay on processing of updated tiles, in seconds. This can reduce potentially expensive re-rendering
# of frequently updated tiles (such as due to machines, pistons, quarries or other automation). Values can
# also be set on individual worlds and individual maps.
tileupdatedelay: 36
# Tile hashing is used to minimize tile file updates when no changes have occurred - set to false to disable
enabletilehash: true
# Optional - hide ores: render as normal stone (so that they aren't revealed by maps)
hideores: true
# Optional - enabled BetterGrass style rendering of grass and snow block sides
better-grass: true
# Optional - enable smooth lighting by default on all maps supporting it (can be set per map as lighting option)
smooth-lighting: false
# Optional - use world provider lighting table (good for custom worlds with custom lighting curves, like nether)
# false=classic Dynmap lighting curve
use-brightness-table: true
# Optional - render specific block IDs using the texures and models of another block ID: can be used to hide/disguise specific
# blocks (e.g. make ores look like stone, hide chests) or to provide simple support for rendering unsupported custom blocks
block-id-alias:
# "14": 1
# "15": 1
# "16": 1
# Default image format for HDMaps (png, jpg, jpg-q75, jpg-q80, jpg-q85, jpg-q90, jpg-q95, jpg-q100)
# Has no effect on maps with explicit format settings
image-format: jpg-q80
# use-generated-textures: if true, use generated textures (same as client); false is static water/lava textures
# correct-water-lighting: if true, use corrected water lighting (same as client); false is legacy water (darker)
# transparent-leaves: if true, leaves are transparent (lighting-wise): false is needed for some Spout versions that break lighting on leaf blocks
use-generated-textures: true
correct-water-lighting: true
transparent-leaves: true
# ctm-support: if true, Connected Texture Mod (CTM) in texture packs is enabled (default)
ctm-support: true
# custom-colors-support: if true, Custom Colors in texture packs is enabled (default)
custom-colors-support: true
# Control loading of player faces (if set to false, skins are never fetched)
fetchskins: true
# Control updating of player faces, once loaded (if faces are being managed by other apps or manually)
refreshskins: true
# Customize URL used for fetching player skins (%player% is macro for name)
skin-url: "http://skins.minecraft.net/MinecraftSkins/%player%.png"
render-triggers:
#- playermove
- playerjoin
- blockplaced
- blockbreak
- leavesdecay
- blockburn
- chunkgenerated
- blockformed
- blockfaded
- blockspread
- pistonmoved
- explosion
#- blockfromto
#- blockphysics
- structuregrow
- blockgrow
#- blockredstone
# Title for the web page - if not specified, defaults to the server's name (unless it is the default of 'Unknown Server')
webpage-title: "ERPL MC Map"
# The path where the tile-files are placed.
tilespath: web/tiles
# The path where the web-files are located.
webpath: web
# The path were the /dynmapexp command exports OBJ ZIP files
exportpath: export
# The network-interface the webserver will bind to (0.0.0.0 for all interfaces, 127.0.0.1 for only local access).
# If not set, uses same setting as server in server.properties (or 0.0.0.0 if not specified)
webserver-bindaddress: 192.168.0.120
# The TCP-port the webserver will listen on.
webserver-port: 8123
# Maximum concurrent session on internal web server - limits resources used in Bukkit server
max-sessions: 16
# Disables Webserver portion of Dynmap (Advanced users only)
disable-webserver: true
# Enable/disable having the web server allow symbolic links (true=compatible with existing code, false=more secure (default))
allow-symlinks: true
# Enable login support
login-enabled: true
# Require login to access website (requires login-enabled: true)
login-required: false
# Period between tile renders for fullrender, in seconds (non-zero to pace fullrenders, lessen CPU load)
timesliceinterval: 0.08
# Maximum chunk loads per server tick (1/20th of a second) - reducing this below 90 will impact render performance, but also will reduce server thread load
maxchunkspertick: 180
# Progress report interval for fullrender/radiusrender, in tiles. Must be 100 or greater
progressloginterval: 800
# Parallel fullrender: if defined, number of concurrent threads used for fullrender or radiusrender
# Note: setting this will result in much more intensive CPU use, some additional memory use. Caution should be used when
# setting this to equal or exceed the number of physical cores on the system.
parallelrendercnt: 3
# Interval the browser should poll for updates.
updaterate: 2000
# If nonzero, server will pause fullrender/radiusrender processing when 'fullrenderplayerlimit' or more users are logged in
fullrenderplayerlimit: 4
# If nonzero, server will pause update render processing when 'updateplayerlimit' or more users are logged in
updateplayerlimit: 8
# Target limit on server thread use - msec per tick
per-tick-time-limit: 10
# If TPS of server is below this setting, update renders processing is paused
update-min-tps: 19.6
# If TPS of server is below this setting, full/radius renders processing is paused
fullrender-min-tps: 19.9
# If TPS of server is below this setting, zoom out processing is paused
zoomout-min-tps: 19.8
showplayerfacesinmenu: true
# Control whether players that are hidden or not on current map are grayed out (true=yes)
grayplayerswhenhidden: true
# Use player permissions to order player list: first to last, players are ordered by first permission listed that they have
# That is, anyone with first listed permission goes before anyone with second, etc, with users with none of the nodes going last
player-sort-permission-nodes:
- bukkit.command.op
# Set sidebaropened: 'true' to pin menu sidebar opened permanently, 'pinned' to default the sidebar to pinned, but allow it to unpin
#sidebaropened: true
# Customized HTTP response headers - add 'id: value' pairs to all HTTP response headers (internal web server only)
#http-response-headers:
# Access-Control-Allow-Origin: "my-domain.com"
# X-Custom-Header-Of-Mine: "MyHeaderValue"
# Trusted proxies for web server - which proxy addresses are trusted to supply valid X-Forwarded-For fields
trusted-proxies:
- "127.0.0.1"
- "0:0:0:0:0:0:0:1"
# Join/quit message format for web chat: set to "" to disable notice on web UI
joinmessage: "%playername% joined"
quitmessage: "%playername% quit"
spammessage: "You may only chat once every %interval% seconds."
# format for messages from web: %playername% substitutes sender ID (typically IP), %message% includes text
webmsgformat: "&color;f[DynMap] %playername%: &color;7%message%"
# Control whether layer control is presented on the UI (default is true)
showlayercontrol: true
# Enable checking for banned IPs via banned-ips.txt (internal web server only)
check-banned-ips: true
# Default selection when map page is loaded
defaultzoom: 2
defaultworld: world
defaultmap: flat
# (optional) Zoom level and map to switch to when following a player, if possible
followzoom: 2
#followmap: surface
# If true, make persistent record of IP addresses used by player logins, to support web IP to player matching
persist-ids-by-ip: true
# If true, map text to cyrillic
cyrillic-support: false
# If true, coordinates will be rounded
round-coordinates: true
# Messages to customize
msg:
maptypes: "Map Types"
players: "Players"
chatrequireslogin: "Chat Requires Login"
chatnotallowed: "You are not permitted to send chat messages"
hiddennamejoin: "Player joined"
hiddennamequit: "Player quit"
# URL for client configuration (only need to be tailored for proxies or other non-standard configurations)
url:
# configuration URL
configuration: "standalone/MySQL_configuration.php"
# update URL
update: "standalone/MySQL_update.php?world={world}&ts={timestamp}"
# sendmessage URL
sendmessage: "standalone/MySQL_sendmessage.php"
# login URL
login: "standalone/MySQL_login.php"
# register URL
register: "standalone/MySQL_register.php"
# tiles base URL
tiles: "standalone/MySQL_tiles.php?tile="
# markers base URL
markers: "standalone/MySQL_markers.php?marker="
# Customization commands - allows scripts to be run before/after certain events
custom-commands:
image-updates:
# Command run just before any image file is written or updated: run with single parameter with fully qualified file name
preupdatecommand: ""
# Command run just after any image file is written or updated: run with single parameter with fully qualified file name
postupdatecommand: ""
# Snapshot cache size, in chunks
snapshotcachesize: 800
# Snapshot cache uses soft references (true), else weak references (false)
soft-ref-cache: true
# Set to true to enable verbose startup messages - can help with debugging map configuration problems
# Set to false for a much quieter startup log
verbose: false
# Enables debugging.
#debuggers:
# - class: org.dynmap.debug.LogDebugger
...
#migrate-chunks: true
Also...
Bukkit.yml:
`connection-throttle: 6000`
Spigot.yml:
`netty-threads: 3`
</details>
*formatting this was a pain*
Understood on the request - I'll need to see how much of a pain it is: the storage system is abstracted, and most of the I/O is off the server thread, but not all of it.
Gotcha - I've made a couple of changes in the latest SNAPSHOT builds on https://dynmap.us/builds/dynmap (SNAPSHOT, not the RC builds). Let me know if you give it a try and how it goes.
I'm going to try out one of the integration modules (you're using the GriefPrevention one) to see if there is something relative to the API use by those mods I might be able to do something about