With a white test texture, rotating the face makes it obvious:

The reason for this is that Vanilla expects the order of vertices to imply where they are in the relation to the cardinal directions of the world. (see net.minecraft.client.renderer.FaceInfo)

While it's not particularly fast, an easy fix is using Vanilla's way of calculating this info. See net.minecraft.client.renderer.block.model.FaceBakery#recalculateWinding

Interestingly, this can even be observed on Fabric.