wanted to travel to an ancher but in the line of sight is an inventory block so instead of traveling i open the inventory
if in line of sight is a normal block in front i can travel

maybe force using of shift to travel to a specific anchor and normal click with the staf does the teleport thing?

This works as intended and there is no plan to change this. However, if you don't care about using travel anchors as elevators, you could disable elevation in your client config and then jump to teleport to prevent opening the inventory.