I wish this mod would come with more in-depth, detailed documentation. Things missing are:

Are the rules in addition to existing spawn rules or instead of? Both should be possible. e.g. spawning witches in swamps but nowhere else, and spawning mooshrooms in mesa's while still on islands?

Also, spawning in water blocks doesn't seem to work. So having squids spawn in oceans but not in other biomes doesn't work?

What mobs have natural spawn-rules and which need to go in potentialspawn? A default list (generated each run) would be nice, just like the mod "tool progression".

Also how do spawn-chances, weights and caps work exactly?

The spawn.json can only be used to alter already existing spawns. You will not be able to add (for example) blaze spawning with that. To add new spawns you have to use potentialspawn.json. This is documented in the wiki.

What rule did you try for the squids?

Okay, to be fair, I was a bit frustrated with this mod because I have been trying to get it to do as I want for over a week now. Sorry.
I know I need to add new spawns to potentialspawn, it is just not always clear which spawns are and which are not on by default, esp. from mods.
The main problem I have is that it is unclear if the rules in spawn replace or add to existing rules. And if they add, how to remove. E.g. only have witches spawn in swamps or squids in oceans, The only example that comes close in the documentation is the one about hostiles only spawing below ground. And that would suggest I need to include a 'deny' rule for all other biomes (over 150). If this is the case, is there some way to add spawn rules in potentialspawn after removing them in the same file?

In regards to squid and water, I gave up on that 3 weeks ago and don't have all the rules I tried anymore.

I'll give an example: In my modpack I have multiple mods that add mobs. One allows me to disable spawning in the config without removing the entities from the game, the other doesn't. So for the former mod I can simply disable spawning in the config and then add en new rul to potentialspawn.json. But for the latter I don't know what the existing rules are (which biomes, does it despawn, height etc.) so I don't know what to remove or add. Ideally I want to remove all rules and then add my own.

@evertx check out the potentialspawn "remove" section here: https://wiki.mcjty.eu/mods/index.php?title=ControlMods#Potential_Spawn

that does what you want, albeit you have to do it per-mob