I dropped a few hours into playing in a new save and made a temporary base in a village I found that spawned over a ravine. I've payed close attention to how they route around and saw no problems involving villagers walking off cliffs BUT I did take care to build walls just in case. So after taking precautions, things were fine until the village was just completely empty besides a single cleric after sleeping in a bed to prevent monsters from spawning.

Edit: A large chunk of villagers were just randomly in the ocean kinda far from the village and after leading them back to the village they just go back to the spot I found them.