You can see a problem on screenshots, my mods folder is:
fabric-api-0.86.0+1.19.4.jar
optifabric-1.13.25.jar
OptiFine_1.19.4_HD_U_I3.jar

Having the same issue on 1.20.1
Optifine_1.20.1_HD_U_I5.jar
fabric-api-0.86.0+1.20.1.jar
optifabric-1.13.25.jar

Same issue on 1.20.1

Its because of Fabric API 0.86.0, if you use Fabric API 0.85.0 it works fine.
I reported same issue on #1123