This may be a just "me error" but can't break item frames and such interaction (ex: quark "totem of holding"), need to rebind the special attack to another key to be able.

If you're using the 4.1.3 version, update it to the latest version since that bug is fixed later.