I just recently updated to 1.16.4 with a server but the game doesn't even get to the loading screen with optifabric/optifine. I removed all mods other than fabric api, optifabric, and optifine, where it still crashed. Removing optifine and optifabric lets it run just fine. The game doesn't even seem to give a crash log so I can't share one.

Try Downgrading fabric API to version [1.16.4] Fabric API 0.28.4
the changes with the latest fabric API version cause the crash