BYG 1.3.5.9
fabric mod loader 0.13.1
IT'S NOT TERRABLENDER BECAUSE I TESTED WITH JUST THAT

First, I really like this mod, I don't want to have to remove this from my mod pack but I can't get it to work for servers at all.

It works fine in client I haven't found any issues in that but for servers I can't join. Remember the fact that this server had the mod in it and was vanilla. Both didn't work as long as the client had BYG.
I looked at the log and there are no lines in there relating to the actual problem.

you didnt provide logs ......

I did a lot of testing and apparently, I had to reset everything, delete configs and install bug fix mods. Delete this thread. I fixed it
My bad...