I find the conflict odd, as Lithium is a server-side mod. It should not affect client-sided rendering, which Optifabric does (I do apologize if the conflict is not Optifabric's fault, and rather Lithium's).

Both Optifabric and Lithium are on version 1.17 and are updated to the latest version.

Optifine optimizes some things that lithium also optimizes. They actually conflict.

Lithium almost works fine on 1.16 (there's a single fix/feature disabled it needs to work), 1.17 I've not tried with yet but it's probably similar. Lithium's changes do influence rendering, the conflict on 1.16 is related to Lithium's changes to optimise chunk storage, it just doesn't target rendering directly so misses the pain OptiFine would otherwise cause