When in game, there don't seem to be any problems until you make a portal. The problem is the same with both nether portals and portals to the twilight forest. Once these portals are within render distance (even when you can't see them) the world will randomly reload/rerender at random intervals. I've never had this issue before with any of the mods I'm currently using and other people on the server are experiencing the same issue. Others on the server do not see the random flashes at the same times as I do. I did read an old issue about a compatibility issue with Serene seasons, but it seems like that was fixed. No mix of optifine settings has changed or stopped this behavior.

I have the most recent version of serene seasons for 1.12.2 from curseforge, if that's not the correct one could you point me to it? So far no other mods have made a difference and removing serene seasons seemed to help.

Upon closer inspection I found that I ended up with an old release of Serene Seasons off of CurseForge. I had version 1.2.16 instead of 1.2.18 which is the latest. After updating it, everything seems to working perfectly. Sorry for the confusion.

That very much sounds like the serene seasons issue, are you sure you've got the fixed version?

It could also be caused by any other mod which does a similar thing (rendering blocks based on some world-global state) while not expecting there to be multiple worlds loaded at the same time (and therefore repeatedly switching between them).
You would probably have to figure out which mod does that by removing mods until it stops and then narrowing it down to just the broken mod. Once you have, report it to them. This almost certainly has to be fixed in the third-party mod.