I think it would be cool and add extra immersion if villages get angry if you damage/steal their property, like if you steal stuff out of the blacksmith chest or start breaking houses' windows (similar to the Mad Villagers mod for 1.12).
Maybe you could buy a building permit or something from the villagers, that let you break blocks in the village boundaries (but they still don't let you open their chests). If you don't have a permit and break a couple blocks, guards give you a warning, but if you keep breaking stuff they'll start attacking you. There could also be a blacklist for certain blocks that they don't care about like, dirt blocks, leaves, grass, etc.

While this sounds cool I have no idea how to determine whats yours and whats part of the village. Automatic building tagging will get removed soon because it causes more errors than it was useful, which makes it even harder to check whether its property or not. Loot chests "may" work because I think I can fetch an event regarding that.

I can offer remarks from villagers nearby, like "I hope you don't vandalize", "What are you building here?". No change in hearts, just interactions.

Closed as deconfirmed, due to the prior reasoning. We simply wouldnt really have a good way to do this, unless @Luke100000 wanted to do a system similar to piglins (Where taking a block such as a bed or chest loot can anger them), but that also can cause false-positives if your simply trying to re-arrange stuff.

TL;DR: It's simply too costly as well as way too complex to fit within scope at this time.