When the mod is installed on the server but not on the client the red X is still shown when looking at a multiplayer server.
It is still possible to join the server.
Implement proper display tests on forge and load the mod only on the server side when on fabric.