I think it would be great if NPCS would invade settlements. Bandits for example could spawn at their castles and camps and invade towns and cities. The only way to prevent this would be to defeat their castles and have a garrison installed