Other issues submitted here have already pointed out how sometimes, spiders can get confused or stand still when they're on the ceiling and can't find a valid path to their target.

This is more in lines of a new feature than a bug fix or pathfinding optimization, but I think it would be helpful and thematic if a spider would release itself from the ceiling and drop down whenever its attacker or target was below it and relatively close by (~16 blocks or fewer).
This would likely require spiders to gain immunity to the Fall damage type, as many other mods have given them. This immunity could have a toggle in the config if desired, for those worried spiders would become too powerful.

If you see a spider on the roof of a cave, and shoot at it with an arrow, it makes sense that it would leap down to attack you instead of continuing to cling there, getting riddled with arrows.
Similarly, the idea of a spider lurking on the ceiling and dropping down onto its unsuspecting prey is not only a genuine tactic for many real spiders, but is a very common fantasy trope that would add to the terror of these creatures.

It's possible this may not work for certain reasons, but I think this would serve a useful purpose of correcting some limitations of 3-dimensional climbing pathfinding while also adding "realistic" and intuitive reactive behavior to spiders.