Once I've placed something, no matter what I do I can't remove it, even when it is completely empty, with no upgrades.. Is there trick I'm missing or am I stuck with a barrel in the middle of my house?

You can't "accidentally" break them in creative mode as they just pop back into place, but you can hold shift to break them. In survival everything should work normally.