I want the guard villager won't fight back when players attacking them, because sometimes player might do it by mistake. But i think they should still fight players that have very low reputation(like -100).
Now it seems that any cases if a player attacks a guard villager it will fight back.

The Friendly Fire mod would be able to mitigate this:
https://www.curseforge.com/minecraft/mc-mods/friendly-fire