When trying to remove dyes/banners from the elytra, it equips the elytra instead. But removing dyes from other items still works fine.

What can I do to fix this? It would save a lot of time if this were possible