version: primitive mobs-1.2.3a
Theoretical behavior: In witch huts, only witches should spawn, however, I observed treasure slimes, haunted tools, this mod's spiders and this mod's skeletons spawning as well.

Is there any particular rule in vanilla that prevents zombies etc from spawning in witch huts? I don't think so honestly. My mobs definitely cannot REPLACE the witch btw.

As far as I know, the way that Vanilla Minecraft works is that every mob that would spawn inside a witch hut is replaced with a witch, so while you're correct that no particular rule prevents zombies/skellies from spawning in a witch hut it's as if they wouldn't, you can easily test this by testing a witch farm.