Some information

Operating system: Windows 11
Java version: Java 16
Minecraft version: 1.17.1 (should be an issue with every version)
Baritone version: artifact from pull request #2850
Other mods (if used): uninteresting (no blocks are being modified or anything what baritone should care about)

How to reproduce

Get map with a block and ladders without a block on the other side (impossible in 1.12.2 without hacking) (the block should be a few blocks at least)
Try to make baritone go up the ladders when the "wrong" side is closer. (baritone always takes the side closer to the start, obviously)

Modified settings

none

Video

I know i modified some settings, but i tested it, they don't make a difference
https://youtu.be/gssID17CLKg

Final checklist

• I know how to properly use check boxes
• I have included the version of Minecraft I'm running, baritone's version and forge mods (if used).
• I have included logs, exceptions and / or steps to reproduce the issue.
• I have not used any OwO's or UwU's in this issue.