If this is entirely a client-side mod, you can use annotations to prevent forge/neoforge versions the mod from crashing when the file is in a dedicated server folder. Additionally:

Fabric > You can also put in fabric.mod.json that the "environment" is "client".

Forge 1.20.1+ > Forge added an entry 'clientSideOnly' which you can include in mods.toml that marks the mod as such, which is nice.