With shaders, both singleplayer and multiplayer, there is a significant (2-3 sec) lag when passing through the portal. Even if passing back and forth (so everything is supposed to be still loaded).
It seems that ALL chunks are loaded again, even though at least the nearest chunks were supposed to be loaded all along.
I took a look at the config file, and ALL render distances are marked as "client-side setting" in the comments. So, where are the server-side settings for how many chunks to keep loaded?..

After thinking about the shader support problem, I've arrived to a certain understanding that I want to confirm. Apparently, shaderpacks use different shaders for different dimensions. Hence, upon passing through the portal, they need to switch shaders, and that requires reloading all chunks. So, the chunks are probably loaded, but they still need to be reloaded after a shader switch, is that correct?

This is Optifine's doing. I don't know exactly what it does since there's no source code for it I can't check either.

A bit of MC terminology:

• Render distance is the distance within which chunks are drawn to the screen and as such client-side by nature (unless it's synchronized between all clients but there's rarely ever a reason to do that).
• View distance is the distance within which chunks are loaded on the server (slightly more complicated in 1.14+) and sent to the clients
  BP portals have dynamic view distance. For most of them, it's the server view distance minus the distance between player and portal minus one chunk (for travelhut it's minus 4 chunks to more quickly limit them).

These shouldn't be related to your issue though, since OF usually isn't installed on the server. I.e. the reload you're seeing is fully client-side. The server doesn't re-send the chunks, it's just the client that re-prepares them for rendering (with shaders they need different preparation than without shaders).

Your understanding of the shader problem is correct.
I don't think a full chunk reload is technically required when coming back to a world where chunks are already loaded but OF just isn't aware of the fact that it's the same world that already was prepared (In a vanilla client, there's only ever one world loaded. So if you go from the overworld into the nether and then back to the overworld, that's now technically a different overworld as far as the client is concerned).

The lag you're seeing is also in part just due to OF loading the new dimension's shaders (which takes time). Unless that's fixed in OF (i.e. unless OF supports having multiple shaders loaded at the same time), there's nothing really I can (or want to) do about it, so I'm closing this issue. Feel free to re-open once (if ever) OF supports that.

I have found the relevant issue in Optifine, so let's link it here: sp614x/optifine#2419