While you're holding an item in your off-hand (for example torches) the construction wand refuses to work.

If you are holding blocks on your off-hand, then wand tries to place those instead of the block you are looking at. This is a feature that allows you to for example build trims in a different material.