I've gathered together my information from the Discord discussion and added some more details.

It looks like there is inventory desync when playing on my server using StorageDrawers-1.19.2-11.1.3 with "invertClick = true".
I've tested with Storage Drawers as the only mod on both server and client.
To recreate the bug I just place a drawer and start using it.

When changing invertClick to true, left-click should now insert a full stack.
Visually the stack disappears from my inventory and hand, but it actually did the default action and removed a single item from the drawer instead.
The stack appears in my inventory again if I try to place something else in that slot.
If I right-click, it visually looks like it removes a single item from the drawer, until I try to place said item and it disappears. Instead I can insert a stack into the drawer using right-click, like when it's not inverted.

About the server:
Docker image: itzg/minecraft-server
Minecraft version: 1.19.2
Forge version: 43.1.23 and 43.2.0
Mods: StorageDrawers-1.19.2-11.1.3

invertClick is working fine in singleplayer.

I would really love for invertClick to work, since it changes a mediocre mod experience to a great one playing Vivecraft on my Quest 2.