Spyglass stays in offhand when the offhand is empty. It should go back to inventory just how it behaves when the offhand is occupied.

Hi! Im unable to reproduce this issue. What version are you using, forge or fabirc? Are you using any mods/modpack? Which one?

I'm on Fabric and Inventory Profiles Next seems to be causing the issue.

Disabling or blacklisting spyglass from auto refill in Inventory Profiles Next fixes this.