If repair mode selects a non-mending armor piece/tool, it will rapidly cycle in and out of the player's off-hand and interrupt the mending process

That sounds like something that is not supposed to happen... when/how exactly does this occur?

I can get a recording in a bit and some screenshots of potentially relevant settings; this is on a PaperMC server, no plugina related to mending when this was observed. Rockets in off-hand, mending sword in main hand (1st hotbar slot), infinity (non-mending) bow or other damaged non-mending item in the second hotbar slot.

Here, got a video uploaded demonstrating the glitch and the items involved. No keys need to be held for the glitch to occur, just pressing the toggle hotkey to enable repair mode.
https://youtu.be/OwXDWiOZSLM

Might be paper protection against that

What is your Lists -> repairModeSlots set to? And which MC/mod version is this in?

Also, which items in your inventory actually have Mending, and are not at full durability? But yeah I think it's likely what vacla said, Paper kicks in due to the fast 3 slot click packets that happen here.

Basically what Tweakeroo tries to do, is swap a mending item to your offhand slot, by using 3 slot swap actions via the hotbar (ie. the same that happens when you press a number key over a slot). It selects the hotbar slot one to the right of you current item, and tries to first swap the found repairable item to that slot, then swap that slot with the offhand slot, and then swap that slot with the original slot the mending item was in. The end result is that the mending item would be swapped with the item in the offhand, without picking up any items to the cursor (because if that action de-syncs then it's annoying that you suddenly have an item in your cursor when you open your inventory).

If I can find the time, i might do another test in singleplayer, though thankfully this'll be a non-issue in 1.16

Tested again in singleplayer, behavior cannot be replicated. Closing this issue.