Minecraft: 1.12.2
Forge: 14.23.5.2824

Mods: 1
yoyos_1.12.2-1.2.2.20

Have nothing in your main-hand and try attacking, you will see that you can punch normally. Now... equip any yoyo in your off-hand and try attacking again. You'll notice that your attacks are slower than usual.

OR

Equip a Stone Axe since it has 0.8 attack speed and try attacking. You will notice that it's slow with attacking. Now... equip any yoyo in your off-hand and try attacking again. You will notice that your axe attacks faster than usual.

In short... the yoyo's attack speed in your off-hand will overwrite your attack speed for your main-hand. This can temporally be fixed if you re-equip the main-hand AFTER you put the yoyo in your off-hand, just don't pause the game as the off-hand attack speed will once again overwrite the main-hand attack speed.

This also happens with any of the Tinker's Construct yoyos if they're in your off-hand.

EDIT
If you un-equip the yoyo in your off-hand and you have a Stone Axe in your main-hand. The Stone Axe's attack speed will be treated as if you had no item held meaning I can attack quickly with the axe. Pausing the game or re-equipping the item will fix the attack speed.