USually I have about 140+ fps, but when i look up of down they go to 60+.
And if I look at fog it goes down to 15-20 fps. I'd say it's un-playable.

Did you try to use OptiFine or something else?

Yes, with and without, and with other fps optimizing mods aswell.
I think main issue is 64 layers of "fog.png" stretched and slightly transparent to mimic fog start. Sure, fog look good, but cause massive fps drop.

The advanced fog rendering difficulty was reduced by 64 times. Version 1.2.4.12.

It's looking much better now, but still get about 20 fps if near fog, I use "Better FPS" mod to make game more playable, but still.

I got couple suggestions to reduce lag. Consider using any of it.

1. When you go down in mist, you should disable (hide, stop, remove) mist layer, because you can't see it above you when you go down to level 60, but you still get fps drop when look up.

2. When you go (fly) higher then mist layer more then +40 blocks (approx level 140), you should replace mist with opaque white plane, you shouldn't keep that mist layer all the time (fade in the white plane, and remove mist)

3. When you walk away from biome "Border", or "_biomename_Border" you should also disable mist, so you don't get fps drop when look at own feet in non "Down" biomes.

4. Disable mist and fog and pollution in non "Down" or "Border" biomes, so you can freely dig down. That would fix ravines filled with poison and mist, especially if you disable mist in biomes far from "Down" and "Border".