I can't find any build that is compatible with 1.20.6 and I get errors out the ass when trying to build it myself with gradle.

I don't think Baritone was updated to 1.20.6 before being updated to 1.21 so no, unless 1.20.5 builds work on 1.20.6 as well there is no working build for 1.20.6.

Actually, just noticed that if you look at the build configuration and forge/fabric/neoforge metadata of the 1.20.5 branch you'll see that it builds against 1.20.6 and 1.20.5 is just an additional supported version.
No idea why the branch name is "1.20.5" and the commit message was "port to 1.20.5".

I can confirm the 1.20.5 build works on 1.20.6 :)