Huge server performance and client latency issues
Sir-Will opened this issue · 13 comments
2018-12-19 22:36:20 | [RAW] Packet | Count | Size
2018-12-19 22:36:20 | [RAW] BQ_NET_CHAN | 136088 | 432.0MB
2018-12-19 22:36:20 | [RAW] SPacketParticles | 42792 | 1.6MB
2018-12-19 22:36:20 | [RAW] chunkpregenerator | 13535 | 277.6KB
2018-12-19 22:36:20 | [RAW] SPacketEntityProperties | 9619 | 425.7KB
2018-12-19 22:36:20 | [RAW] SPacketUpdateHealth | 8925 | 87.2KB
2018-12-19 22:36:20 | [RAW] Packet | Count | Size
2018-12-19 22:36:20 | [RAW] BQ_NET_CHAN | 136088 | 432.0MB
2018-12-19 22:36:20 | [RAW] SPacketChunkData | 1129 | 40.8MB
2018-12-19 22:36:20 | [RAW] SPacketParticles | 42792 | 1.6MB
2018-12-19 22:36:20 | [RAW] SPacketUpdateTileEntity | 5520 | 1.2MB
2018-12-19 22:36:20 | [RAW] FML|HS | 34 | 993.9KB
This seems to also cause connection and latency issues.
@Funwayguy is it intended that sendToAll instead of sendToPlayer is being used here?
@Sir-Will Have you tested these solutions? Did you find any other fix to this issue?
So I just wanted to add that not sure how your getting the cool packet data but with 10 people online our server is spiking none stop destroying my bandwidth on the server.
We are doing about 20 Mbps with 10 people online.
If I idle in fresh generated chunks with a view distance of 6 and no chunk updates my client will do from 1 Mbps all the way to 3 Mbps spiking.
I should be doing 0.1-0.2 at most in the same conditions.
There seems to be some improvement but not a ton when the Router actually shows the bandwidth / spikes. Not sure how @Sir-Will got the packet info so well.
https://i.imgur.com/jdU1hjQ.png
One thing I noticed in game instead sitting 1.0 to spike to 2.0 I sit at 0.1 and spike to 1.0 now. But this is still with nothing around, etc. I would need to learn another tools / methods to actually monitor the packets to be sure.
If @Sir-Will can tell me what he used I will attempt to replicate it.
In the current implementation you get:
for every online player every 3 sec for every active quest that does not have uncomplete tasks of fluid,location,meeting,retrieval,scoreboard,xp sent this quests full config (name,tasks,rewards,etc) plus the progress of every player to every online player.
In bytes a simple quest is 2KB in config and 400b progress bones+~100 bytes for every player completed and about 70 bytes for players doing this quest.
With this, let's say a server has 10 online users who have 10 different active crafting quests which 100 persons have completed we end up with about 120KB every player every data sent to 10 users 10 times. Which totals to 1.2MB download per player over 3 sec and 12MB upload over 3 sec for server which is an average of 3200Kbit/s download for users and 32Mbit/s upload for server.
Obviously you won't see this drastic bandwidth usage since my calculations uses a extreme case of 10 active crafting or hunting quests when in many packs the most common task type is retrieval. But then again, if you're a public server who has seen 200 or more players you will get these statistics even with 5 active quests and so on. Or using a modpack like GT:NH where you can easily have 20 active crafting quests.
In addition to this, the calculations here are done only for one situation, you should also consider the whole database being sent on player logins (to that player) which will happen more often than not when players with lower bandwidth get frustrated with lag and try to relogin. On that player login the name cache of every player ever recorded also gets sent to everyone online. With an old server this will be massive.
The fix i am proposing in #512 is addressing every single point i outlined here. It is by no means a code-wise good-looking solution but in my opinion the current situation is worse and therefore should be seriously considered.
Resolved in #512
Feel free to test it thru https://minecraft.curseforge.com/projects/better-questing/files/2651557 and https://minecraft.curseforge.com/projects/better-questing-standard-expansion/files/2651558
I wouldn't go as far as saying it's outright resolved but this is an experimental solution which may help until a more permanent code revision is made.
In the current implementation you get:
for every online player every 3 sec for every active quest that does not have uncomplete tasks of fluid,location,meeting,retrieval,scoreboard,xp sent this quests full config (name,tasks,rewards,etc) plus the progress of every player to every online player.
In bytes a simple quest is 2KB in config and 400b progress bones+~100 bytes for every player completed and about 70 bytes for players doing this quest.
With this, let's say a server has 10 online users who have 10 different active crafting quests which 100 persons have completed we end up with about 120KB every player every data sent to 10 users 10 times. Which totals to 1.2MB download per player over 3 sec and 12MB upload over 3 sec for server which is an average of 3200Kbit/s download for users and 32Mbit/s upload for server.Obviously you won't see this drastic bandwidth usage since my calculations uses a extreme case of 10 active crafting or hunting quests when in many packs the most common task type is retrieval. But then again, if you're a public server who has seen 200 or more players you will get these statistics even with 5 active quests and so on. Or using a modpack like GT:NH where you can easily have 20 active crafting quests.
In addition to this, the calculations here are done only for one situation, you should also consider the whole database being sent on player logins (to that player) which will happen more often than not when players with lower bandwidth get frustrated with lag and try to relogin. On that player login the name cache of every player ever recorded also gets sent to everyone online. With an old server this will be massive.
The fix i am proposing in #512 is addressing every single point i outlined here. It is by no means a code-wise good-looking solution but in my opinion the current situation is worse and therefore should be seriously considered.
+1
For all the tasks in the Standard Expansion, their writeProgressToNBT functions ignore the users parameter. So they still end up saving progress for every player, even when requested to save progress for specific players. For example, here's TaskRetrieval doing that.
My apologies if I commented in the wrong place. I was looking at your new code, because I want to extend the InvSync plugin to synchronize player progress between different nodes of a multiverse.
You can't see the change anymore since Funwayguy/StandardQuestingPack@e5e3c73 reverted my changes from the main branch and Funawayguy hasn't uploaded the exp_net branch to github like it is for the BetterQuesting repository.
I'm moving stuff across piece by piece rather than all at once. This also makes it much more contained if something breaks especially considering this involves critical questing functionality.
I am aware the parameters are currently ignored but that is because I'm not currently ready to start rewriting that part. The first part was simply getting all those methods put into place and reworking the quest cache. Second (which I'm working on now) is reducing the amount of sync packets being sent from the server side by leveraging the new cache. Then I'll split the data within each request to only those relevant players.
The experimental branch is more of a test run of the concept but not a permanent implementation. I do not recommend using it as a reference for the final version.
About that branch on SE, that's my bad. I forgot to push my local version to remote.
EDIT: Done
https://github.com/Funwayguy/StandardQuestingPack/tree/experimental_net
In the current implementation you get:
for every online player every 3 sec for every active quest that does not have uncomplete tasks of fluid,location,meeting,retrieval,scoreboard,xp sent this quests full config (name,tasks,rewards,etc) plus the progress of every player to every online player.
In bytes a simple quest is 2KB in config and 400b progress bones+~100 bytes for every player completed and about 70 bytes for players doing this quest.
With this, let's say a server has 10 online users who have 10 different active crafting quests which 100 persons have completed we end up with about 120KB every player every data sent to 10 users 10 times. Which totals to 1.2MB download per player over 3 sec and 12MB upload over 3 sec for server which is an average of 3200Kbit/s download for users and 32Mbit/s upload for server.Obviously you won't see this drastic bandwidth usage since my calculations uses a extreme case of 10 active crafting or hunting quests when in many packs the most common task type is retrieval. But then again, if you're a public server who has seen 200 or more players you will get these statistics even with 5 active quests and so on. Or using a modpack like GT:NH where you can easily have 20 active crafting quests.
In addition to this, the calculations here are done only for one situation, you should also consider the whole database being sent on player logins (to that player) which will happen more often than not when players with lower bandwidth get frustrated with lag and try to relogin. On that player login the name cache of every player ever recorded also gets sent to everyone online. With an old server this will be massive.
The fix i am proposing in #512 is addressing every single point i outlined here. It is by no means a code-wise good-looking solution but in my opinion the current situation is worse and therefore should be seriously considered.
Maybe, problem it's resolved for storage raid?