Apparently the world keeps spawning Adventurers at the Inn even with the chunk unloaded (considering no one was playing on the server for a few days) and they also don't despawn after 2 days.
since there are a lot of them (600+) the server is not very happy it doesn't exactly have the most powerful CPU on the market

How to Reproduce
Open a server/lan, create a village with an Inn and go out to do something with its chunk not loaded

They also remain there even after the Inn is removed from the village.

Version
MCA 7.4.8 MC 1.18.2 Forge