Hello - currently (and this may be an old issue) it appears that abilities on toolbars, whose buttons have been bound to mouse buttons, cannot be used to target abilities while mousing over grid2 frames.

Example:
With evoker's "Verdant Embrace", if resident on a bar whose buttons are bound to 1-9, or alt+Letter (alt-A, alt-S, alt-D), and using bliz's new built-in mouseover casting, you can mouse over a grid2 target, cast, and target the moused-over person, as expected.

If that ability is moved to a bar where the bound mapping is Mouse Button 4 or 5 (or shift+ Mouse 4, or 5) - the ability can still be cast with "no target", or with a traditional click-to-set-target, but the button does not respond when mousing over grid2 frames. When mousing over players in the world, the mouse targeting works.

Version 2.0.72 Stable.

Also tested with traditional "Mouse over macro" config, with the same result.

Grid2 does not manage mouse over casting, so i can do nothing to fix this issue. Anyway i cannot reproduce the issue, i did a test with several chaman spells, assigned to mouseover-button1/button2/button4/button5 using blizzard clickcasting interface, and moved the same spell to an action bar button assigned to mouse button4 or button5, and it was working fine.
Update:
I use options/keybindings/clickcasting to configure mouse4/mouse5 buttons, maybe you are speaking of the options/combat/mouseover cast option, but this option only works for keys, not for mouse buttons, this does not work even with blizzard frames (blizzard player/party/raid frames).
If you want to configure mouseover button4/button5 goto:
Options/keybinding/clickcasting and link button4/button5 to the desired spells.

Ok thank you for your help! I didn't think to test with blizzard frames - I havn't used them in years because I use Grid2 instead :)

It definitely sounds like an issue for bliz. In the meantime I will try the clickcasting config. Thank you!