Sounds like a little problem, but its really annoying and visible.

Optifine
shaders/no shaders - no diffrence
no orther performance affecting mods

I can't seem to be able to reproduce your problem. Can you send me the log file of a run where the problem is happening?

Just don't use optifine. Use the GLSL shaders mod and
better biome blend and if you're using shaders and if you are not,
use Phosphor, Lithium and better biome blend fabric.