Dynmap-Forge/Fabric

Dynmap-Forge/Fabric

888k Downloads

Option for local MySQL requests be async

ice-fly opened this issue · 12 comments

commented

I've been experiencing server lag... I am running paper-298 (currently) and have been trying to track down why I'm seeing lag spikes.
After many timings reports & a suspicion of Dynmap, I set the write interval on the JsonFileClientUpdateComponent writeinterval to 35

Sure enough, every 35 seconds my server chugs...

[13:13:33 INFO]: Server tick times (avg/min/max) from last 5s, 10s, 1m:
[13:13:31 INFO]: ◴ 1.7/0.3/93.9, 1.2/0.3/93.9, 0.9/0.2/94.5
[13:13:48 INFO]: ◴ 0.7/0.2/2.0, 0.7/0.2/2.3, 0.9/0.2/94.5
...
[13:14:05 INFO]: ◴ 0.7/0.3/2.1, 0.7/0.3/2.1, 0.8/0.2/93.9
[13:14:06 INFO]: ◴ 1.7/0.3/93.8, 1.2/0.3/93.8, 0.9/0.2/93.9

I suspect this issue is often not as easy to detect as most people are running MySQL on their machine. I have a tunnel setup to a web-host that runs the SQL database & dynmap standalone. Hence, my access times are longer & why I have a noticeable slowdown to my server.
Also, disconnects from jdbc make server unusable due to constant re-connect requests.

The fix to this is to move dynmap's database operations off the main thread & make them asynchronous.

commented

Reading between the lines on #2775 could it be that when localhost is the target address, the plugin expects fast access times? And thus doesn't bother with async?

commented
Timings Logo This needs updated to do async... https://github.com/webbukkit/dynmap/blob/v3.0/DynmapCore/src/main/java/org/dynmap/JsonFileClientUpdateComponent.java
commented

I think getConnection() needs async:
An error log... git-Paper-318 (MC: 1.15.2) :

Error
[02:05:29 ERROR]: Current Thread: Server thread
[02:05:29 ERROR]:       PID: 25 | Suspended: false | Native: false | State: WAITING
[02:05:29 ERROR]:       Stack:
[02:05:29 ERROR]:               [email protected]/java.lang.Object.wait(Native Method)
[02:05:29 ERROR]:               [email protected]/java.lang.Object.wait(Unknown Source)
[02:05:29 ERROR]:               org.dynmap.storage.mysql.MySQLMapStorage.getConnection(MySQLMapStorage.java:526)
[02:05:29 ERROR]:               org.dynmap.storage.mysql.MySQLMapStorage.getStandaloneFile(MySQLMapStorage.java:989)
[02:05:29 ERROR]:               org.dynmap.JsonFileClientUpdateComponent.handleWebChat(JsonFileClientUpdateComponent.java:373)
[02:05:29 ERROR]:               org.dynmap.JsonFileClientUpdateComponent$1.run(JsonFileClientUpdateComponent.java:152)
[02:05:29 ERROR]:               app//org.bukkit.craftbukkit.v1_15_R1.scheduler.CraftTask.run(CraftTask.java:84)
[02:05:29 ERROR]:               app//org.bukkit.craftbukkit.v1_15_R1.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:461)
[02:05:29 ERROR]:               app//net.minecraft.server.v1_15_R1.MinecraftServer.b(MinecraftServer.java:1256)
[02:05:29 ERROR]:               app//net.minecraft.server.v1_15_R1.DedicatedServer.b(DedicatedServer.java:430)
[02:05:29 ERROR]:               app//net.minecraft.server.v1_15_R1.MinecraftServer.a(MinecraftServer.java:1173)
[02:05:29 ERROR]:               app//net.minecraft.server.v1_15_R1.MinecraftServer.run(MinecraftServer.java:962)
[02:05:29 ERROR]:               [email protected]/java.lang.Thread.run(Unknown Source)
Can't keep up! Is the server overloaded? Running 31989ms or 639 ticks behind

I'd rather dynmap render slower than chew up my main thread and cause tps lag

commented

OK - just to be clear - your scenario is using MySQL as a storage type, and running with an external server. I'm doing some problem determination now, and I'm seeing calls to the storage from the server thread when words are activated, but I haven't found anything else yet. Are you using any other plugins that might be calling the Dynmap APIs for, say, markers or the like? Just trying to find the cases to be pinned down here - straight up Spigot + Dynmap with external web doesn't seem to be making storage calls from the server thread, except on what would be relatively rare cases.

commented

Yes, I have a particularly egregious case: The sql server has ping times of ~100ms. (Shoestring mc server budget) I'm using a webhost who provides me a shared webserver to host server website, sql database, & dynmap standalone files.
I'm doing a full render, but cpu never goes above 50% utilization
Dynmap version: core=3.0-RC2-318, plugin=3.0-RC2-318 (just realized I'm out of date, updating now)

Plugins (22) This seemed to happen before Dynmap-GriefPrevention, snowdrift, & snowier world were installed. Not that it isn't them... (just less likely) AsyncWorldEdit, AutoRestart*, ChopTree2, ClaimslistClassifier, DiscordSRV, dynmap*, Dynmap-GriefPrevention*, Essentials, EssentialsAntiBuild, EssentialsChat, EssentialsProtect, EssentialsSpawn, GriefPrevention, LuckPerms, MineableSpawners, OreAnnouncer, ProtocolLib, SnowDrift*, SnowierSnow, Vault, WorldEdit, WorldGuard
git-Paper-317
Config Dynmap: ``` # All ... hires
        user 'userid' password 'password' and table prefix 'prefix')
         type: mysql
         hostname: localhost
         port: 24306
         database: (Obfuscated)
         userid: (Obfuscated)
         password: (Obfuscated)
         prefix: ""
         flags: "?allowReconnect=true"
         
       components:
         - class: org.dynmap.ClientConfigurationComponent
         ...
         - class: org.dynmap.JsonFileClientUpdateComponent
           writeinterval: 6
           sendhealth: false
           sendposition: true
           allowwebchat: true
           webchat-interval: 12
           hidewebchatip: true
           includehiddenplayers: false
           use-name-colors: true
           use-player-login-ip: true
           require-player-login-ip: true
           block-banned-player-chat: true
           hideifshadow: 8
           hideifundercover: 8
           hideifsneaking: true
           # Require login for web-to-server chat (requires login-enabled: true)
           webchat-requires-login: true
           # If set to true, users must have dynmap.webchat permission in order to chat
           webchat-permissions: true
           # Limit length of single chat messages
           chatlengthlimit: 250
           hide-if-invisiblity-potion: true
           hidenames: false
          
         - class: org.dynmap.SimpleWebChatComponent
           allowchat: true
           # If true, web UI users can supply name for chat using 'playername' URL parameter.  'trustclientname' must also be set true.
           allowurlname: false
         
         # Note: this component is needed for the dmarker commands, and for the Marker API to be available to other plugins
         - class: org.dynmap.MarkersComponent
           type: markers
           showlabel: true
           enablesigns: true
           # Default marker set for sign markers
           default-sign-set: playerMarkers
           # (optional) add spawn point markers to standard marker layer
           showspawn: true
           spawnicon: world
           spawnlabel: "Spawn"
           # (optional) layer for showing offline player's positions (for 'maxofflinetime' minutes after logoff)
           showofflineplayers: false
           offlinelabel: "Offline"
           offlineicon: offlineuser
           offlinehidebydefault: true
           offlineminzoom: 2
           maxofflinetime: 120
           # (optional) layer for showing player's spawn beds
           showspawnbeds: false
           spawnbedlabel: "Spawn Beds"
           spawnbedicon: bed
           spawnbedhidebydefault: true
           spawnbedminzoom: 0
           spawnbedformat: "%name%'s bed"
           # (optional) show world border (vanilla 1.8+)
           showworldborder: true
           
         - class: org.dynmap.ClientComponent
           type: chat
           allowurlname: false
         - class: org.dynmap.ClientComponent
           type: chatballoon
           focuschatballoons: false
         - class: org.dynmap.ClientComponent
           type: chatbox
           showplayerfaces: true
           messagettl: 8
           # Optional: set number of lines in scrollable message history: if set, messagettl is not used to age out messages
           scrollback: 80
           # Optional: set maximum number of lines visible for chatbox
           visiblelines: 8
           # Optional: send push button
           sendbutton: true
         - class: org.dynmap.ClientComponent
           type: playermarkers
           showplayerfaces: true
           showplayerhealth: false
           # If true, show player body too (only valid if showplayerfaces=true
           showplayerbody: false
           # Option to make player faces small - don't use with showplayerhealth
           smallplayerfaces: true
           # Optional - make player faces layer hidden by default
           hidebydefault: false
           # Optional - ordering priority in layer menu (low goes before high - default is 0)
           layerprio: 0
           # Optional - label for player marker layer (default is 'Players')
           label: "Players"
           
         #- class: org.dynmap.ClientComponent
         #  type: digitalclock
         - class: org.dynmap.ClientComponent
           type: link
           
         - class: org.dynmap.ClientComponent
           type: timeofdayclock
           showdigitalclock: true
           showweather: true
         # Mouse pointer world coordinate display
         - class: org.dynmap.ClientComponent
           type: coord
           label: "Location"
           hidey: false
           show-mcr: false
           
         # Note: more than one logo component can be defined
         #- class: org.dynmap.ClientComponent
         #  type: logo
         #  text: "Dynmap"
         #  #logourl: "images/block_surface.png"
         #  linkurl: "http://forums.bukkit.org/threads/dynmap.489/"
         #  # Valid positions: top-left, top-right, bottom-left, bottom-right
         #  position: bottom-right
       
         - class: org.dynmap.ClientComponent
           type: inactive
           timeout: 1800 # in seconds (1800 seconds = 30 minutes)
           redirecturl: inactive.html
           showmessage: 'You were inactive for too long.'
         
         #- class: org.dynmap.TestComponent
         #  stuff: "This is some configuration-value"
       
       # Treat hiddenplayers.txt as a whitelist for players to be shown on the map? (Default false)
       display-whitelist: false
       
       # How often a tile gets rendered (in seconds).
       renderinterval: 0.64
       
       # How many tiles on update queue before accelerate render interval
       renderacceleratethreshold: 64
       
       # How often to render tiles when backlog is above renderacceleratethreshold
       renderaccelerateinterval: 0.32
       
       # How many update tiles to work on at once (if not defined, default is 1/2 the number of cores)
       tiles-rendered-at-once: 2
       
       # If true, use normal priority threads for rendering (versus low priority) - this can keep rendering
       # from starving on busy Windows boxes (Linux JVMs pretty much ignore thread priority), but may result
       # in more competition for CPU resources with other processes
       usenormalthreadpriority: false
       
       # Save and restore pending tile renders - prevents their loss on server shutdown or /reload
       saverestorepending: true
       
       # Save period for pending jobs (in seconds): periodic saving for crash recovery of jobs
       save-pending-period: 960
       
       # Zoom-out tile update period - how often to scan for and process tile updates into zoom-out tiles (in seconds)
       zoomoutperiod: 8
       
       # Control whether zoom out tiles are validated on startup (can be needed if zoomout processing is interrupted, but can be expensive on large maps)
       initial-zoomout-validate: true
       
       # Default delay on processing of updated tiles, in seconds.  This can reduce potentially expensive re-rendering
       # of frequently updated tiles (such as due to machines, pistons, quarries or other automation).  Values can
       # also be set on individual worlds and individual maps.
       tileupdatedelay: 36
       
       # Tile hashing is used to minimize tile file updates when no changes have occurred - set to false to disable
       enabletilehash: true
       
       # Optional - hide ores: render as normal stone (so that they aren't revealed by maps)
       hideores: true
       
       # Optional - enabled BetterGrass style rendering of grass and snow block sides
       better-grass: true
       
       # Optional - enable smooth lighting by default on all maps supporting it (can be set per map as lighting option)
       smooth-lighting: false
       
       # Optional - use world provider lighting table (good for custom worlds with custom lighting curves, like nether)
       #   false=classic Dynmap lighting curve
       use-brightness-table: true
       
       # Optional - render specific block IDs using the texures and models of another block ID: can be used to hide/disguise specific
       #  blocks (e.g. make ores look like stone, hide chests) or to provide simple support for rendering unsupported custom blocks
       block-id-alias:
       #    "14": 1
       #    "15": 1
       #    "16": 1
         
       # Default image format for HDMaps (png, jpg, jpg-q75, jpg-q80, jpg-q85, jpg-q90, jpg-q95, jpg-q100)
       # Has no effect on maps with explicit format settings
       image-format: jpg-q80
       
       #  use-generated-textures: if true, use generated textures (same as client); false is static water/lava textures
       #  correct-water-lighting: if true, use corrected water lighting (same as client); false is legacy water (darker)
       #  transparent-leaves: if true, leaves are transparent (lighting-wise): false is needed for some Spout versions that break lighting on leaf blocks
       use-generated-textures: true
       correct-water-lighting: true
       transparent-leaves: true
       
       # ctm-support: if true, Connected Texture Mod (CTM) in texture packs is enabled (default)
       ctm-support: true
       # custom-colors-support: if true, Custom Colors in texture packs is enabled (default)
       custom-colors-support: true
       
       # Control loading of player faces (if set to false, skins are never fetched)
       fetchskins: true
       
       # Control updating of player faces, once loaded (if faces are being managed by other apps or manually)
       refreshskins: true
       
       # Customize URL used for fetching player skins (%player% is macro for name)
       skin-url: "http://skins.minecraft.net/MinecraftSkins/%player%.png"
       
       render-triggers:
         #- playermove
         - playerjoin
         - blockplaced
         - blockbreak
         - leavesdecay
         - blockburn
         - chunkgenerated
         - blockformed
         - blockfaded
         - blockspread
         - pistonmoved
         - explosion
         #- blockfromto
         #- blockphysics
         - structuregrow
         - blockgrow
         #- blockredstone
       
       # Title for the web page - if not specified, defaults to the server's name (unless it is the default of 'Unknown Server')
       webpage-title: "ERPL MC Map"
       
       # The path where the tile-files are placed.
       tilespath: web/tiles
       
       # The path where the web-files are located.
       webpath: web
       
       # The path were the /dynmapexp command exports OBJ ZIP files
       exportpath: export
       
       # The network-interface the webserver will bind to (0.0.0.0 for all interfaces, 127.0.0.1 for only local access).
       # If not set, uses same setting as server in server.properties (or 0.0.0.0 if not specified)
       webserver-bindaddress: 192.168.0.120
       
       # The TCP-port the webserver will listen on.
       webserver-port: 8123
       
       # Maximum concurrent session on internal web server - limits resources used in Bukkit server
       max-sessions: 16
       
       # Disables Webserver portion of Dynmap (Advanced users only)
       disable-webserver: true
       
       # Enable/disable having the web server allow symbolic links (true=compatible with existing code, false=more secure (default))
       allow-symlinks: true
       
       # Enable login support
       login-enabled: true
       # Require login to access website (requires login-enabled: true)
       login-required: false
       
       # Period between tile renders for fullrender, in seconds (non-zero to pace fullrenders, lessen CPU load)
       timesliceinterval: 0.08
       
       # Maximum chunk loads per server tick (1/20th of a second) - reducing this below 90 will impact render performance, but also will reduce server thread load
       maxchunkspertick: 180
       
       # Progress report interval for fullrender/radiusrender, in tiles.  Must be 100 or greater
       progressloginterval: 800
       
       # Parallel fullrender: if defined, number of concurrent threads used for fullrender or radiusrender
       #   Note: setting this will result in much more intensive CPU use, some additional memory use.  Caution should be used when
       #  setting this to equal or exceed the number of physical cores on the system.
       parallelrendercnt: 3
       
       # Interval the browser should poll for updates.
       updaterate: 2000
       
       # If nonzero, server will pause fullrender/radiusrender processing when 'fullrenderplayerlimit' or more users are logged in
       fullrenderplayerlimit: 4
       # If nonzero, server will pause update render processing when 'updateplayerlimit' or more users are logged in
       updateplayerlimit: 8
       # Target limit on server thread use - msec per tick
       per-tick-time-limit: 10
       # If TPS of server is below this setting, update renders processing is paused
       update-min-tps: 19.6
       # If TPS of server is below this setting, full/radius renders processing is paused
       fullrender-min-tps: 19.9
       # If TPS of server is below this setting, zoom out processing is paused
       zoomout-min-tps: 19.8
       
       showplayerfacesinmenu: true
       
       # Control whether players that are hidden or not on current map are grayed out (true=yes)
       grayplayerswhenhidden: true
       
       # Use player permissions to order player list: first to last, players are ordered by first permission listed that they have
       # That is, anyone with first listed permission goes before anyone with second, etc, with users with none of the nodes going last
       player-sort-permission-nodes:
         - bukkit.command.op
         
       # Set sidebaropened: 'true' to pin menu sidebar opened permanently, 'pinned' to default the sidebar to pinned, but allow it to unpin
       #sidebaropened: true
       
       # Customized HTTP response headers - add 'id: value' pairs to all HTTP response headers (internal web server only)
       #http-response-headers:
       #    Access-Control-Allow-Origin: "my-domain.com"
       #    X-Custom-Header-Of-Mine: "MyHeaderValue"
       
       # Trusted proxies for web server - which proxy addresses are trusted to supply valid X-Forwarded-For fields
       trusted-proxies:
         - "127.0.0.1"
         - "0:0:0:0:0:0:0:1"
         
       # Join/quit message format for web chat: set to "" to disable notice on web UI
       joinmessage: "%playername% joined"
       quitmessage: "%playername% quit"
       
       spammessage: "You may only chat once every %interval% seconds."
       # format for messages from web: %playername% substitutes sender ID (typically IP), %message% includes text
       webmsgformat: "&color;f[DynMap] %playername%: &color;7%message%"
       
       # Control whether layer control is presented on the UI (default is true)
       showlayercontrol: true
       
       # Enable checking for banned IPs via banned-ips.txt (internal web server only)
       check-banned-ips: true
       
       # Default selection when map page is loaded
       defaultzoom: 2
       defaultworld: world
       defaultmap: flat
       # (optional) Zoom level and map to switch to when following a player, if possible
       followzoom: 2
       #followmap: surface
       
       # If true, make persistent record of IP addresses used by player logins, to support web IP to player matching
       persist-ids-by-ip: true
       
       # If true, map text to cyrillic
       cyrillic-support: false
       
       # If true, coordinates will be rounded
       round-coordinates: true
       
       # Messages to customize
       msg:
           maptypes: "Map Types"
           players: "Players"
           chatrequireslogin: "Chat Requires Login"
           chatnotallowed: "You are not permitted to send chat messages"
           hiddennamejoin: "Player joined"
           hiddennamequit: "Player quit"
       
       # URL for client configuration (only need to be tailored for proxies or other non-standard configurations)
       url:
           # configuration URL
           configuration: "standalone/MySQL_configuration.php"
           # update URL
           update: "standalone/MySQL_update.php?world={world}&ts={timestamp}"
           # sendmessage URL
           sendmessage: "standalone/MySQL_sendmessage.php"
           # login URL
           login: "standalone/MySQL_login.php"
           # register URL
           register: "standalone/MySQL_register.php"
           # tiles base URL
           tiles: "standalone/MySQL_tiles.php?tile="
           # markers base URL
           markers: "standalone/MySQL_markers.php?marker="
       
       # Customization commands - allows scripts to be run before/after certain events
       custom-commands:
           image-updates:
               # Command run just before any image file is written or updated: run with single parameter with fully qualified file name
               preupdatecommand: ""
               # Command run just after any image file is written or updated: run with single parameter with fully qualified file name
               postupdatecommand: ""
       
       # Snapshot cache size, in chunks
       snapshotcachesize: 800
       # Snapshot cache uses soft references (true), else weak references (false)
       soft-ref-cache: true
               
       # Set to true to enable verbose startup messages - can help with debugging map configuration problems
       # Set to false for a much quieter startup log
       verbose: false
       
       # Enables debugging.
       #debuggers:
       #  - class: org.dynmap.debug.LogDebugger
       ...
       #migrate-chunks: true

Also...
Bukkit.yml:
`connection-throttle: 6000`
Spigot.yml:
`netty-threads: 3`
</details>
*formatting this was a pain*
commented

Understood on the request - I'll need to see how much of a pain it is: the storage system is abstracted, and most of the I/O is off the server thread, but not all of it.

commented

Gotcha - I've made a couple of changes in the latest SNAPSHOT builds on https://dynmap.us/builds/dynmap (SNAPSHOT, not the RC builds). Let me know if you give it a try and how it goes.

commented

On first glance. It's looking like times have been cut in half...

Server tick times (avg/min/max) from last 5s, 10s, 1m:
? 6.1/3.8/67.8, 5.8/3.8/67.8, 6.1/3.5/67.8
Timings timings

Thanks!
You're good to close this, or I will if status is still good in a week.

commented

Let's keep it open for now - see where we are next weekend.

commented

I'm going to try out one of the integration modules (you're using the GriefPrevention one) to see if there is something relative to the API use by those mods I might be able to do something about

commented

Issue seems resolved for now

commented

Still solved. Thanks Mr. Primm!