Version: GrimoireOfGaia3-1.12.2-1.6.0-beta3
I have found many red circles in journey map and try to figure out what happened. This is what I see before I die. I am pretty sure they are not wandering from other biomes.

They can spawn in the Beach biome.

Mobs who spawn in the Beach biome always spawn underground though. If the red circles where actually in the island and not around it then that is an issue.

Which mobs spawn there? Is it the Ocean mobs? Underground mobs? Or both?

They can suddenly spawn in the center (underground) of my mushroom island basement when I travel back. Also even from this screenshot you can see that (under the mini map) this biome is MushroomIsland but not beach or MushroomIslandShore (which should also be peaceful).

That means Ocean mobs are spawning in underground the MushroomIsland biome.

I will add more restrictions to their spawn conditions in the next update.

Double checked and tested. I guess they are spawned in MushroomIslandShore because this biome also has a biome type of BEACH.
(And why can they spawn in the middle of a mushroom island? Because there are several tiny sized MushroomIslandShores inside the island, even hundred meters to the nearest ocean...)

Thanks for your help and hard working for this mod. Looking forward for next update!

Minecraft biome generation has a few issues. The River biome for example can be found in areas where there is no water even present.

The fix will be implemented in 1.6.0-beta5.

I have only found Mermaid and Sahuagin yet. So it is possible both.

Fixed in 1.6.0-beta5.