I want to create a group with a cap of five entities, but they are tameable, so if I tame all of them and keep them with me anytime would this group keep spawning more entitites until it reach ten, or the entities I tamed should be killed so more would be allowed to spawn?

Tames / nondespawnable entities are counted. They would need to be killed / unloaded to not count.

Is there any way to change that behavior through the config files?

Not at this time.

Thanks anyway.