Please post a crash log. We can't help you if you we don't have a crash log.
That said, there is a known incompatibility between Immersive Portals and Sodium. If you have Sodium installed, please try to replicate this without Sodium.

I have tried many configurations using Sodium, Lithium, And Phosphor with Immersive portals and I will share all the outcomes I have gotten.

Any Configuration with Sodium crashed with this exit code:

The game crashed whilst initializing game
Error: java.lang.RuntimeException: Could not execute entrypoint stage 'client' due to errors, provided by 'imm_ptl_core'!
Exit Code: -1

Crash Reports:

Sodium+Lithium+Phosphor:
crash-2021-02-22_18.36.05-client(Sodium+Lithium+Phosphor).txt

Sodium+Lithium:
crash-2021-02-22_18.37.16-client(Sodium+Lithium).txt

Sodium+Phosphor:
crash-2021-02-22_18.38.07-client(Sodium+Phosphor).txt

Here are the Results for my test with Phosphor and Lithium:

Phosphor: Immersive Portals Worked well and no matter what I did, I found no bugs or glitches.

Phosphor+Lithium: Same Results from Phosphor.

Lithium: Same Results from Phosphor.

Additional Information: I am using the Latest Fabric API and Mod loader.

So after these test I'm pretty sure that the problem was from Sodium because no matter what I did I could not replicate the crashes without Sodium. I hope this helps and if you need additional information ill do my best to give it to you.

Any Configuration with Sodium crashed with this exit code

So then why are you posting this on Phosphor's issue tracker? From your own testing this would indicate a conflict with Sodium. Regardless, the issue is explained if you read Immersive Portal's project page.