when you just come back to the village there is that whole greeting session right. When using the waypoint stone mod and the greeting thing is happening all the villagers teleports with you to the other location

That sounds like a waystone issue tbh (or a mod adding "compatibility")