See the ending of #1823 for insight to the issue
Further insight with another tester has highlighted the following, and is believed to be a cache issue and is currently the only issue holding back a release:

CC: @Brhsoftco as the other user experimenting with this

The issue predates to v1.2.10.
v1.2.9 is without issue
I also encountered similar issues with #farm in 1.6.1, but it needs more investigation to know if it's related.
UPDATE
farming doesnt respond to the same context behaviour. Im going to investigate further and make a new thread after

Updating title, this is a more severe issue

@CDAGaming I mentioned it in the other thread. When you make a collision with the portal and end up going to the nether, baritone won't find your portal and seemingly keeps trying to path (even though it already ended up at the right block). Instead, it will path to a really long distance away until you tell it to stop and then immediately reissue the command. In this instance, it correctly paths back to your portal until you collide once again.

@CDAGaming @Dominikaaaa
It might be worthwhile to create some sort of cache verification. E.g. make sure that parameters match before trying to access the cache; like if the current dimension ID is the same as the cached chunk, etc. It might be worthwhile to even append extra parameters to cached information, like the dimension ID it was cached in, and the biome ID. Before baritone tries to load the cache from memory, it should check the cached dimension ID against the current one, and then flush the memory cache if it's not equal. If it is equal, then it may proceed to load the cache. Maybe this is something? I'm still trying to figure out the structure of baritone's build system, as I've never worked with Gradle before.

Update: Investigations with @leijurv have pointed out the extendCacheOnThreshold which has yielded positive results in v1.2.10 as well as 1.16.1.

However, as it's settings note denounces, it is resource intensive and immediately tanks the fps within 1.16.1 (Even for a 1060 and 8th Gen i7 with 32GBs of RAM).

Further investigations pending closer to 1.16.2s Release Date will be scheduled to locate a solution that would have the best of the cache and WorldScanner working in harmony as it did in 1.2.9.

However, as it's settings note denounces, it is resource intensive and immediately tanks the fps within 1.16.1 (Even for a 1060 and 8th Gen i7 with 32GBs of RAM).

tfw i thought baritone couldn't affect my fps even more, it's already low

Oh trust me, the option while it does work, significantly slows down my dev environment

Here is the code as a reference for it's effect:

        locs = prune(ctx, locs, filter, max, blacklist, dropped);

        if (!untracked.isEmpty() || (Baritone.settings().extendCacheOnThreshold.value && locs.size() < max)) {
            locs.addAll(WorldScanner.INSTANCE.scanChunkRadius(
                    ctx.getBaritone().getPlayerContext(),
                    filter,
                    max,
                    10,
                    32
            )); // maxSearchRadius is NOT sq
        }

It might be worthwhile to create some sort of cache verification. E.g. make sure that parameters match before trying to access the cache; like if the current dimension ID is the same as the cached chunk, etc. It might be worthwhile to even append extra parameters to cached information, like the dimension ID it was cached in, and the biome ID. Before baritone tries to load the cache from memory, it should check the cached dimension ID against the current one, and then flush the memory cache if it's not equal. If it is equal, then it may proceed to load the cache. Maybe this is something?

That's not really how the cache works, you can take a look at the source code to learn more.

the best of the cache and WorldScanner working in harmony as it did in 1.2.9.

It isn't this simple. The use case of placing a portal in the world (while there are NO portals ANYWHERE beforehand), then goto-ing it, is a reasonably fast test to run, but it doesn't line up so well with "real world" (in quotes) baritone usage.

So obviously, between v1.2.9 and v1.2.10 is logan's massive changes to commands and such. He spent a TON of time benchmarking worldscanner and making it faster.

Looking through the changes, this is just getting deeper and deeper the more I look into it. Logan removed it entirely then added it back? (this check in rescan)

Anyway, v1.2.9 did not respect extendCacheOnThreshold in the special case of 0. While the setting is supposed to control any value less than the maximum (which is 64).

If worldscanner performance was actually degraded then that's it's own issue, but that very much does not line up with the testing that occurred at that time.

The cache exists for a reason lol

Yea, while i originally wasn't looking at the FPS, it was present and having more of an impact on performance especially on the 1.16.1, but I am unsure if it is exclusively bc of that option or related to something else, which I'd plan we could investigate after 1.16.2 releases.

Closed as Resolved, will be available in next release!