You can easily hit a villager on accident in a crowded village. Plus, Following guards will be damaged by you.

One way to compensate this, is perhaps a forgiveness implementation, where either leaving a village for some time OR by gifting something to reduce the aggression will clear aggro (If you cant tell, I've been playing a lot of skyrim lately)

Guards will warn you the first hit and after that will only beat you up depending on the damage you dealt. Unlike golems btw who will fight you even after death. What behavior do you expect?

Yeah they do warn you, but when you trying to frantically swinging your sword when fighting along them them it is bound to happen eventually, plus, not mentioning the pets, who will maul the target and cause and war between the two forces, it is better to not be able to hit them at all.

What if it's based off of the player's reputation in the village? We could forgive more hits before attacking if the player's reputation is higher or above a threshold. Each hit will reduce hearts anyways so reputation will decrease if they keep hitting villagers, eventually getting a reaction.

I added a combination of the ideas here: guards allow one more hit for every 30 hearts (configurable) as well as decrease your hit count every minute. This should make them far less aggressive.