Investigating - Serversync file syncing issues.
P3rf3ctXZer0 opened this issue · 74 comments
Serversync Version: serversync-2.6.22.jar
Minecraft Version: 1.12.2 - 1.12.2 - 14.23.5.2838
Issue:
I am getting users saying they cannot connect
"this is my current config."
Configuration file
general {
B:PUSH_CLIENT_MODS=true
}
serverconnection {
I:SERVER_PORT=38067
}
rules {
S:CONFIG_INCLUDE_LIST <
>
S:DIRECTORY_INCLUDE_LIST <
config
customdisc
guides
local
mods
pixelmon
scripts
>
S:FILE_IGNORE_LIST <
config/Discord-Integration.cfg
mods/s-dcintegration-1.0.1-b.jar
mods/s-Tiquality-FAT-1.12.2-GAMMA-1.5.3-r.jar
>
}
misc {
S:LOCALE=en_US
}
Client log looks like SS never managed to get to the connection phase.
Though the server log seems to suggest the client in question timed out on their connection for some reason.
Is this a constant failure, or intermittent?
I forgot one key piece of information - when multiple people use serversync is when the issue occurs most often (I think like 3 people.)
Interesting, I'll look into the timeout. It should be one instance per client but it might be reaching outside itself.
I just had a simple thought ... what if instead of placing serversync config in config folder it is placed in a separate config called ssconfig so you can also configure it to completely not press the config to clients - I think part of the reason it is having issues is because I think it scans itself at one point when it should not - ever.
A lot of mod depend on config files matching in forge and fabric so by just adding the "config" folder I noticed some errors - problem is they are erratic.
Another thing that might help is wildcards for folders so a great example of a folder that has multiple child folders is Pixelmon Reforged. The issue is Serversync only like scanning the first layer and does not go deeper and this throws client side errors hard. Even when explicitly told to.
It was also still pressing files that I explicitly told it not to do. Also it seems to trip out after 192,000 files or more but I am going to study this in great detail today.
Watching the log it deleted things with extensions it didn't understand. Maybe add a way to add accepted extensions or additional accepted extensions.
example
.abc <- This just a random example
.hocon <- pixelmon reforged uses this.
.twk <- inventory tweaks uses this.
Okay so I just checked the users folder and ss is completely ignoring the PUSH CLIENT MODS setting
@rheimus this is a huge problem.
I tried changing the file name to not include any special characters and it still would not push that folder.
ServerSync has a feature for clients that allows them to refuse client mods from a server.
Has the client in question set "REFUSE_CLIENT_MODS" to true in their serversync-client.cfg?
ServerSync only parses one type of file, that being '.jar' files.
If SS finds a .jar file it will look inside it for a minecraft mod information file, if it does not find one then it just treats the file like any normal raw file.
There should be no reason for specific file types to fail (outside of configuration issues) as SS just transfers the file byte for byte across the socket.
Could you send me a file tree dump or screenshot of your minecraft directory along with the serversync-server configuration file being used?
Tree dump commands.
Windows
dir /s > file-tree-dump.txt
Linux
ls -R > file-tree-dump.txt
file-tree-dump.txt
Is this what you wanted?
serversync.zip
Please ignore the ignore list - during testing that was not actually there.
This was how the config list looked like during testing
https://gist.github.com/P3rf3ctXZer0/81afbc9c999057a012249f2976293bd9
I wanted to let you know I found out that after syncing the same files myself 3 times I started failing to get files too. Maybe have a ip sanity check for client as well as for server that is why I felt unique file/folder check system felt best - rather than pure file names. Maybe generate anmd5hash ID per folder? So basically if server-sync detects that any file does not match what is pushed to a folder have it delete any files that do match and re-download them - then do another sanity check. If the sanity is valid write and md5 number and move on to the next folder. It should never fail to recheck files at least once. It should also check for filesize and name changes. If it just checks name or simply checks if true as it does - that would allow a type of failure and would allow users to tamper with file in a way a would could be malicious. I believe too that if you have a server log that keeps - the IPV4s with a single md5checksum or some identifier this too would allow ss to make sure all the same requested files are sent. Try it once delete all the files multiple times and on a remote ip have someone else get those files and you will see what I am talking about. Serversync after a bit starts to believe it sent those files - even though it didn't - like a scatter brained secretary. Hell mimic my setup and grab my mod-pack.
SS
PORT: 38067
IP 209.252.169.81
MODPACK SHELL MULTIMC IMPORTABLE INSTANCE ServerSync.zip
That shell is pre-configured to be able to run my pack after grabbing files. Please grab all files 3 times and then try and actually play. So grab files - Delete files - Grab Files - Delete Files and try to play. It may be an error with multiple ips so I will happily try twice on my end.
I believe if you see first hand what I am experiencing you get a much greater understanding of what is going wrong since clearly we are also in different time-zones. If you want to talk here is my discord ID - simply friend me. P3rf3ctXZer0 aka Zer0Necros#1549
Um I forgot to add things from file tree print but this is the current client mods directory.
On the left is the server including file being pushed and on the right is the client launching after a long update process. Relaunching actually does not have it get the client-side files.
ctm should not be a client mod if it is a required dependency of chisel.
Client mods were designed to be mods that are completely optional, i.e. you can connect to the server without having them present. (NEAT, Optifine, etc)
Your client config has also set REFUSE_CLIENT_MODS=true, this is telling ServerSync to not sync anything that comes from the servers 'clientmods' directory.
SS currently does file comparison in the following sequence per file (all three must pass).
- Do the file names match (files are not equal if this fails) (this one is probably pointless as we do a file hash check anyway)
- Do the files exist in the same location (files are not equal if this fails)
- Do the files have matching content hashes (files are not equal if this fails)
SS does two phases of sync.
- Check configured files from the server, overwrite/copy to the client if they are different
- Sweep the servers configured directories for files that do not exist on the server, delete them if the user has not configured them to be explicitly ignored.
I was interested in setting up some auto testing, the multiple sync's failing would be a decent test case.
Might look into serenity or something.
Okay so ctm cannot be a universal mod as it crashes the server - I know stupid and that does not absolve the whole client folder failing to push either though. As for "serenity" I have no idea what that is.
Don't worry about my rambling 😊, serenity is an auto testing framework / pattern for software (http://www.thucydides.info/#/). Automated testing helps developers make changes with confidence that what they have built works as expected and that nothing else has broken due to the change that has been made. It also lets you test things that are unrealistic or horrible for manual testing, like run the program 1000 times and expect that the same action happens each time.
The client folder is not being transferred due to the client config telling it not to. To transfer 'clientmods' you need the following pair of configuration settings:
Server
PUSH_CLIENT_MODS=true
Client
REFUSE_CLIENT_MODS=false
I'll need accurate data to be of much assistance, the above configurations were set up in a way that would manifest exactly what you are seeing.
Could you provide a zip of your server instance, I can not currently reproduce the issue locally.
https://www.dropbox.com/s/xww9ahjxgy1wthg/mc-1.12.2-FromTheAshes-ss.zip?dl=0
I have to say again though - for best raw expirence just use server sync on that pack I gave you. I did update a few files before giving them to you - only to match the newest pack release ... I doubt you can even find the error locally - do you have a far away server and ip you are using for testing?
Please consider using this #134 (comment)
Besides - testing from a users perspective will give you the most raw experience.
I didn't do it ... - maybe after a failed attempt to sync - does it a have a fallback routine?
That picture is directly off the remote server host. But when I opened up that pack I gave you in #134 (comment) it said ip was set to 127.0.0.1 instead of the ip you see in that image ...
Cool this time it didn't send the configs for some mods ...
Right Side is Server Config
Left Side is Client Config when obtained by serversync.
Its trivial but annoying the server allows you to see FlatColoured Blocks in JEI but the client version was set to ignore - so they don't match. Let me check and see how many configs failed to send. Okay so good news it was the one config ... but the fact that it is not sending all configs means it will break in different ways at some point.
Indeed, you are synchronizing the config folder. The serversync configs are not currently protected so they will be deleted/modified unless explicitly ignored in the ignore lists.
That was the plan, though currently the ignore pattern is wrong.
It is ignoring serversync*, but that will only ignore anything it finds at the top level directory.
Okay - that sounds better - the other thing is make wild cards work so we don't need to manually set deeper folders.
I think unless user explicitly sets it should count recoursive subfolders as things it should mimic too.
That would break the spec for glob patterns, recursive search is defined by '**' double star.
Ps I am staying up soley to help you - so when you have a build to test let me know immediately cause I am getting tired.
So when a user sets add "config/myconfig/**" that makes all recursive folders included as well?
Yeah. Though glob patterns are only supported at the moment in the ignore lists. So you would generally do something more like:
"**/*.disabled"
To ignore anything with the extension disabled in any folder.
The directory include list is recursive by default.
Oh well maybe do me and the world a favor and comment out and add an explination and usage instructions <- I have no memory so essentially you could end up reteaching me 80x's
Would you be shocked if I said I barely understand the explanations in the wiki?
Look at it this way - pretend you are teaching code to a 8 year old. - They have a general feel but are easily distracted by big words and lengthy sketches also easily intimated by things they don't get.
How long will it take to make the code alterations I am getting really tired.
There is a link to a live playground 'globtester' which lets you interactively test exactly which patterns do what.
Though I could probably add story style documentation in there.
There is unlikely to be any changes to test for a while, I have to try and reproduce the problem first.
Sadness ... I think I may have done legit all I can ... farewell friend ... please let me know if I can help again ... night.
There is an early access peek at the new file sync available in my fork if you are interested, note that it does not currently support clientmods: https://github.com/rheimus/ServerSync/releases
As a part of the new file sync I also added in a retry phase. If for whatever reason one of the files fails to sync the client will try a second time.
Nice - recursive retries :D I just moved so for a bit I cannot really test anything.
All good, it will be in alpha for a while until I can get some proper automated end to end testing set up for it. There are almost certainly bugs in the new process that I have not catered for.
Yeah - hopefully sometime after the 15th I will have what I need to test things again.