Lag caused by chunks loading
andrewkm opened this issue · 18 comments
HolographicDisplays is currently taking up over 25% of our server tick.
https://i.imgur.com/o4qOc6P.png
HolographicDisplays::Event: c.g.f.h.l.MainListener (ChunkLoadEvent)
I've seen several other servers report the same issues.
Now sure what's causing it.
We are running Paper 1.13.2 Build 639
@filoghost probably a placeholder task going crazy? :/
Can someone provide the link to the timings report, or even better, a CPU timings snapshot (such as with VisualVM)? The CPU snapshot would help a lot. I failed to reproduce on my machine, loading and unloading as fast as possible a chunk with 1000 holograms in it, each one with a placeholder.
@sgdc3 Could be a coincidence, but I didn't know it compiles a regex each time internally. It may have some effects on performance, so the custom name is now cached in 1.13+ 66666dc
@filoghost If you'd like to join the MythicMobs Discord and chat with users there, many of them have been having this issue and can probably either help you reproduce it or take the logs for you. Here's an invite link: https://discord.io/mythicmobs
Is this getting fixed @filoghost
How many holograms do you have? Please try without other plugins if it still occurs.
We’ve of course tested with other hologram plugins as well as with no holograms.
This specific plugin is the one that hits performance really hard as can be seen by the timings.
We had about 30-40 holograms that were in a chunk that was constantly being passed through, (loaded and unloaded) as such there is some pretty big slowdown in that regard and how this plugin handles the loading of holograms.
Keeping the chunk loaded or removing the amount of holograms improves the performance but it’s still no where near as good as it should be.
Same issue here.
It got especially worse after updating to the latest 1.14.4 development build, HolographicDisplay alone was dragging our server down to 10/9 tps.
After disabling the plugin through PlugMan, tps went back at 20.
https://gyazo.com/8e0826cf2100e1f7e56b8e0d0e47052e
https://gyazo.com/bda6e073456bc19f20b04db19c328fd9
(Server version: git-Paper-161 (MC: 1.14.4)
I experienced the same problem and so i did some investigating, it appeared that invisible armorstands would begin to spawn in certain areas, at one point i had over 5000 spawn in one little area and another 3000 in another, once you restart they disappear
I found a fix for my problem, if yours is similar it may help, the area I had holograms I used a flag from world guard to keep the chunk loaded, it fixed my problem at least
v2.4.0. is still affected. Degraded server performance (10 tps) after a couple of hours uptime.
[12:12:50] [Paper Watchdog Thread/ERROR]: The server has not responded for 37 seconds! Creating thread dump
[12:12:50] [Paper Watchdog Thread/ERROR]: ------------------------------
[12:12:50] [Paper Watchdog Thread/ERROR]: Server thread dump (Look for plugins here before reporting to Paper!):
[12:12:50] [Paper Watchdog Thread/ERROR]: ------------------------------
[12:12:50] [Paper Watchdog Thread/ERROR]: Current Thread: Server thread
[12:12:50] [Paper Watchdog Thread/ERROR]: PID: 26 | Suspended: false | Native: false | State: RUNNABLE
[12:12:50] [Paper Watchdog Thread/ERROR]: Stack:
[12:12:50] [Paper Watchdog Thread/ERROR]: java.util.regex.Pattern$SliceNode.study(Pattern.java:3951)
[12:12:50] [Paper Watchdog Thread/ERROR]: java.util.regex.Pattern$Node.study(Pattern.java:3419)
[12:12:50] [Paper Watchdog Thread/ERROR]: java.util.regex.Pattern.group0(Pattern.java:2947)
[12:12:50] [Paper Watchdog Thread/ERROR]: java.util.regex.Pattern.sequence(Pattern.java:2053)
[12:12:50] [Paper Watchdog Thread/ERROR]: java.util.regex.Pattern.expr(Pattern.java:1998)
[12:12:50] [Paper Watchdog Thread/ERROR]: java.util.regex.Pattern.compile(Pattern.java:1698)
[12:12:50] [Paper Watchdog Thread/ERROR]: java.util.regex.Pattern.<init>(Pattern.java:1351)
[12:12:50] [Paper Watchdog Thread/ERROR]: java.util.regex.Pattern.compile(Pattern.java:1028)
[12:12:50] [Paper Watchdog Thread/ERROR]: java.lang.String.replaceFirst(String.java:2178)
[12:12:50] [Paper Watchdog Thread/ERROR]: org.bukkit.craftbukkit.v1_13_R2.util.CraftChatMessage.fromComponent(CraftChatMessage.java:193)
[12:12:50] [Paper Watchdog Thread/ERROR]: org.bukkit.craftbukkit.v1_13_R2.util.CraftChatMessage.fromComponent(CraftChatMessage.java:162)
[12:12:50] [Paper Watchdog Thread/ERROR]: com.gmail.filoghost.holographicdisplays.nms.v1_13_R2.EntityNMSArmorStand.getCustomNameNMS(EntityNMSArmorStand.java:190)
[12:12:50] [Paper Watchdog Thread/ERROR]: com.gmail.filoghost.holographicdisplays.placeholder.PlaceholdersManager.updatePlaceholders(PlaceholdersManager.java:368)
[12:12:50] [Paper Watchdog Thread/ERROR]: com.gmail.filoghost.holographicdisplays.placeholder.PlaceholdersManager.access$100(PlaceholdersManager.java:36)
[12:12:50] [Paper Watchdog Thread/ERROR]: com.gmail.filoghost.holographicdisplays.placeholder.PlaceholdersManager$1.run(PlaceholdersManager.java:90)
[12:12:50] [Paper Watchdog Thread/ERROR]: org.bukkit.craftbukkit.v1_13_R2.scheduler.CraftTask.run(CraftTask.java:84)
[12:12:50] [Paper Watchdog Thread/ERROR]: org.bukkit.craftbukkit.v1_13_R2.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:449)
[12:12:50] [Paper Watchdog Thread/ERROR]: net.minecraft.server.v1_13_R2.MinecraftServer.b(MinecraftServer.java:1010)
[12:12:50] [HikariPool-1 housekeeper/WARN]: [AntiVPN] {c.z.h.p.HikariPool} HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m6s692ms236µs492ns).
[12:12:50] [Paper Watchdog Thread/ERROR]: net.minecraft.server.v1_13_R2.DedicatedServer.b(DedicatedServer.java:439)
[12:12:50] [Paper Watchdog Thread/ERROR]: net.minecraft.server.v1_13_R2.MinecraftServer.a(MinecraftServer.java:940)
[12:12:50] [Paper Watchdog Thread/ERROR]: net.minecraft.server.v1_13_R2.MinecraftServer.run(MinecraftServer.java:837)
[12:12:50] [Paper Watchdog Thread/ERROR]: java.lang.Thread.run(Thread.java:748)
[12:12:50] [Paper Watchdog Thread/ERROR]: ------------------------------
[12:12:50] [Paper Watchdog Thread/ERROR]: --- DO NOT REPORT THIS TO PAPER - THIS IS NOT A BUG OR A CRASH ---
[12:12:50] [Paper Watchdog Thread/ERROR]: ------------------------------```
I found a fix for my problem, if yours is similar it may help, the area I had holograms I used a flag from world guard to keep the chunk loaded, it fixed my problem at least
Which flag?
Was not easy to find a reason, but it looks like setting precise-hologram-movement to false in the configuration of HD could fix the problem. Can anyone confirm? Sorry if it took so long.
Was not easy to find a reason, but it looks like setting
precise-hologram-movementtofalsein the configuration of HD could fix the problem. Can anyone confirm? Sorry if it took so long.
Thank you! Testing this. Will get back to you once it's been running for a while.
Tested with latest dev build and precise-hologram-movement = false. Seems to be fixed. Thank you.
After some CPU profiling I am 99% sure precise-hologram-movement is the cause. It uses a method that can be slow if called frequently (see #16). I will need to rewrite that part without using stacktraces, using ProtocolLib if necessary.
I pushed a change to reimplement the functionality of precise-hologram-movement through ProtocolLib, fixing the lag problems. Since the option was enabled by default, many users should see big performance improvements if Holographic Displays was slowing their server.
Please test the latest dev build: https://ci.codemc.org/job/filoghost/job/HolographicDisplays