One of the features of the wand of the forest is to rotate blocks around (like chests). However, when I sneak-right-click on the chest it both rotates but also changes the mode of the wand. I was under the impression that this should only happen when sneak rightclicking in the air.

I'm using the latest version (Botania r1.9-342) on Minecraft 1.11.2.

and you confirmed the key bind was not set for mode change?

Botania has no keybinds. The check needs to be revised.