Crafting a bed still seems to be quite a ways away for me, but I am already being swarmed by 2-3 phantoms every night.

Sure, but this is not realy a problem 'cause phantoms does little harm and could be easily slained. It drop very useful material to refit tools.