While swap almost broken tools is on sometimes tool (in my case pickaxe) the durability of the other tool that I'm switching to either decreases to 20 (default durability that the tool swithes) or to full.

This makes no sense... does the item get switched on the server side before the use action finishes and it damages the wrong tool, or what? It's not just an inventory de-sync? Does it persist if you re-log?