As the title states, it doesn't work. Using LegitMine setting and mining ancient debris causes baritone to pathfind in a random direction, even going AWAY from the specified Y level I set legitmine to go to and then walks directly into lava.

I assume the way legit mine would work would be to mine straight (strip mine) until it finds the specified block

Legit mine remembers your starting position (and maybe sometimes resets that, not sure) and tries walking away from there, abusing the pathfinder heuristic to prefer, not enforce, the configured y level. If walking away from the remembered position is easier at a different y level it will use that.

Walking away from the specified y level is expected if there is lava in the way. It shouldn't walk into lava though, unless you have assumeWalkOnLava enabled.

I assume the way legit mine would work would be to mine straight (strip mine) until it finds the specified block, but when I use it for ancient debris it goes up to the surface of the nether for seemingly no reason and goes back down, never staying at the specified y level for legitmine

No you guys maybe got it wrong or maybe I got it wrong, but every time I used legit mine in overworld it just went looking for caves and mining openly visible ores, it never tried to strip mine at all, I'm assuming it doesn't do a good job at ancient debris because it's trying to find it exposed, but there aren't any
I suggest you use the tunnel command for creating tunnels and then do legit mine so that it mines any exposed ancient debris you might not have seen
Since netherack is easy to break you can do a 5x3xN relatively fast, or even just clear a huge area entirely

Or you can use baritone to build a tnt tunnel schematic, either way you will work less than intended and look legit for the most part

Look at my previous comment for what legitMine does. As @figloalds correctly observed that behavior favors finding openly visible ores, which does not work well for deepslate.