Interestingly this appears to only happen outside of dev for me. Might be a difference in configs or might also be indirectly fixed by one of the other third-party mods prevent in dev. Needs further investigation.

Found the reason this doesn't happen in the dev env: It's a case of MinecraftForge/ForgeFlower#11 in ClippingHelper.isBoxInFrustum.