not completely sure this is fixable. but would be nice if it only had to be on the server

I think the only solution is another config option "force-client-mod", which makes the mod a need while a player tries to connect to the server.