I was playing with the Rats mod and ended up getting a piece of post-netherite, post-boss gear within 10 minutes of a fresh survival playthrough because a Rare Feral Ratlantean apothic boss spawned in the overworld and I cheesed it in the ocean.

Can there be a config that blacklists/whitelists certain mobs from becoming bosses? How are modded boss loot tables and boss choices calculated in this mod?

This is a modpack configuration issue. They have explicitly allowed that boss to spawn with that gear in the overworld, only vanilla mobs are supported at base.

I am the one creating this modpack, and I hadn't touched the config for apoth aside from raising the invading boss cooldown. Does Rats in particular have some weird interop for some reason? All modded invading bosses I've seen have come from mods made by Alex (i.e. Rats and Alex's mobs).

This is a modpack configuration issue. They have explicitly allowed that boss to spawn with that gear in the overworld, only vanilla mobs are supported at base.

Either rats has innate compat with Apotheosis (I doubt it), or you have an addon mod adding this functionality, likely Apotheotic Additions if I had to guess. If you want to fix this, either make a datapack to override the boss / gear set file to remove it, or remove the addon that has this behavior.

I do in fact have apoth additions, ig I'll go ask them to add a whitelist/blacklist (or config in general, for that matter) and make a datapack in the meantime.

btw some of the docs in the config are outdated.