Seems we're not done just yet, 1.1.2 broke loading the real chunks
Probably related to the fix of #19 but I feel like it's a separate issue

Forgot to mention this happens only on multiplayer servers, including my own server with low ping
And the mods used are: Sodium snapshot, fabrics api and bobby

Oh, my bad, should have really tested it before publishing.