When on single player it works, but on server, i can't sneak click to remove anything from any of the farms.

I see nothing in the Server Logs.

Figured it out, It wont let me when my offhand is carrying something. If this is a feature, close issue. thanks.

Just for your information, I added an exception for the villager blocks, so that your off hand item isn't considered when sneak-clicking on the blocks.
It is available in the latest version.