Villagers may not be removed from the list if crashed or otherwise incorrectly removed.
A fix would be a timeout, e.g. last seen flag.

Also seems like dead villagers are never removed from a residence.