After testing this a lot I can somewhat confidently say this is a shader issue.

Here is during the loading screen:

And here is after:

And these are my changed shader settings:

Sidenote: It says I changed ManualExposureValue, but this is not the problem as I have ManualExposure disabled

I forgot to mention my Minecraft version is 1.20.1, and I'm using Iris version 1.6.17 + Sodium version 0.5.8