Any way to reduce internal traffic?
aaa8881 opened this issue ยท 4 comments
Description
I'm having issue with internal traffic with around 1200 ~ 1300 playercount atm.
Currently, outbound traffic is around at 500mbps , and internal traffic is hitting 930mbps which is almost at limit of 1Gbps line.
Which is casuing packet loss - obviously very bad sign.
Internal traffic from packend bukkit server to proxy should obviously include outbound traffic ( chunk and stuff ) , so this means that around 400mbps is accounted of plugin messaging.
I found that LP is constantly syncing permission over network to ensure permission syncroniczation.
in this playerbase, can LP generate large traffic from plugin messaging? If then, is there way to reduce it?
I looked into plugin messaging part, and tried to change some settings, but it didn't work out as I expected.
Reproduction Steps
no particular step
Expected Behaviour
no particular behavior
Server Details
pufferfish-61 ( 1.19.3 fork )
LuckPerms Version
5.4.79
Logs and Configs
no log needed I guess
Extra Details
No response
Have you actually confirmed that LP is the cause of this traffic? LP is (supposed to be) quite light on the messaging traffic, only sending a message when permissions actually change. When a message is sent, it's only a small "hey, something changed involving this user/group/track" type message, not the actual change. The backend database is the source of truth for permission state.
In addition, LP defaults to using our sql messaging implementation, which uses a database table as a way of sending these messages. While we do have a implementation for plugin message channels (called pluginmsg in the config), you would have had to explicitly change the messaging service to pluginmsg for it to be used.
Hi. Thanks for reply and no i havent confirmed. Thats one thing i wanna confirm as well.
I saw in /lp verbose on , there are literally thousands of sync-ish looking thing happening every seconds. Does it not indicate plugin messaging? ( real question not rhetorical )
I need to check which is generating excessive traffic for sure? But i cant think of anything that would send that much
While LP is indeed handling hundreds or thousands of permission checks a second on even smaller servers, all of those are handled locally. Each LP instance is responsible for calculating the result of a permission query on their instance. Player data gets loaded on player join, and LP's messaging is only a "hey, you need to refresh this player/group/track from the storage backend", plus usually an associated audit log message to send to any players with log notifications enabled.
While it's not impossible LP is the cause of your excessive data, I find it exceedingly unlikely. I'm not sure if a tool exists that can analyze the traffic to determine the source, but if not I'd suggest binary searching your plugins to find the culprit that way.