Uh... not sure. Seems to happen when the reputation increases. Something with... gossiping? Maybe? No idea. Could totally be a mod conflict, but figured I'd share in case it's not.

Vanilla bug: https://bugs.mojang.com/browse/MC-252486

I'm in the late stages of porting this mod to 1.21+, and I've implemented a manual bugfix for this only for that version.