An easy way to fix this would be

1. Get the distance to the hitbox we're interacting with
2. Get the distance to the best raytrace to the block we're looking at
3. Only perform immersive interaction if 1 < 2.