(Mods being used.)
yoyos_1.12.2-1.2.0.18
Mantle-1.12-1.3.1.21
TConstruct-1.12.2-2.9.1.65

Happens with all the Yo-Yos whether it be vanilla Yo-Yos Tinker's Construct Yo-Yos or the all powerful Rainbow Yo-Yo, etc.

The damage from Yo-Yos when equipped in the Off-Hand will be ignored but instead it goes based off your Main-Hand damage. Say... you have a Rainbow Yo-Yo which has insane damage and such and you equip that in your Off-Hand and nothing in your Main-Hand and you start using it. It'll deal damage as if you were punching with your fist.

If you wield a Diamond Sword alongside the Rainbow Yo-Yo. It's damage will be based off the Diamond Sword.

If into Tinker's Construct... you made a Manyullyn Yo-Yo and a Manyullyn Broadsword and you use the Yo-Yo in the Off-Hand and the Broadsword in the Main-Hand. The Yo-Yo's damage will use the Broadsword's damage which is weaker than with just using the Yo-Yo in your Main-Hand.

The Yo-Yo's in the Main-Hand however... work as intended.