Even a short path cuts my framerate in half, can't really explain why. Well not quite in half, but significant drop. Like 60 to 45.

It was the crazy allocation of tons of objects during Baritone's tick. Wasn't actually anything slow about the render hook code or anything. Thanks to YourKit that is pretty much entirely eliminated.