So I have been using baritone to mine ores but an issue has been that since each ore has the optimum Y-level to mine it in I would get to the Y-level then start up baritone however after about 5 minuts it sort of moves to the least optimal position

For example, I start mining ancient debris at y=15, but 5 minutes later it drifts to y=90 and starts running around for the few debris at the level even when most of the debris at y=15 is not mined out yet. If I do lapis ore(normal not deepslate type) it starts going towards y=0 instead of y=64

Are there any settings in baritone for which to help this tendancy?
I appreciate it if someone can help

You can use minYLevelWhileMining/maxYLevelWhileMining to restrict which ores it will target (not where it will walk; also note that the former is an offset from the lower build limit) and legitMineYLevel to configure which y level it tries to reach when no ore positions are known and exploreForBlocks or legitMine is enabled (if neither is enabled it fails).

As long as there are ores left, and you don't want to outright forbid everything outside a certain range, there is nothing you can do about the drift. It will just walk to whichever block pathfinding reached first.

You can make it more likely for baritone to walk on y level set by legitMineYLevel by

1. lowering blockBreakAdditionalPenalty - Keeping penalty low allows the bot to mine through walls to the next cave. If this is high it will prefer going above ground or through caves to avoid mining.
2. increasing costHeuristic - Allows the bot to consider more costly paths like mining through lots of blocks. If this is low it will prefer going above ground or through caves to avoid mining.

Best advice is to just try out some numbers and see what works best for your situation.

I find that in most cases blockBreakAdditionalPenalty can be set to 12.0 and costHeuristic to 4.0

Note: These settings to make it more likely for the bot to get stuck underground without pickaxe and then you would need to mine with a hand to the surface for wood to make new one.

You should definitely set blockBreakAdditionalPenalty to 0 before starting to increase costHeuristic.
The former changes the meaning of "optimal" to no longer include "break few blocks" while the latter straight up allows suboptimal paths if they are "straighter".

Though noting that there probably won't be much places where it can sprint, 4 should still yield mostly optimal paths (walking is ~4.6 ticks per block). Just remember turning it back down when going to the surface or your paths might end up 10% longer than needed.