I feel like the swords are way too common considering how powerful they are compared to diamond and iron swords, was looting a village earlier and found one in the first 5 minutes of starting my world and it feels like the swords should be significantly more rare.

That is likely the issue.
I'd recommend tweaking the config values to get a loot rate that best matches your modpack.

What version are you using? They shouldn't even be appearing in villages by default.

I have quite a few village mods as well to be fair, and is more likely an issue with that mod not using village loot tables, but I am using 1.50.1.