Version Information

I noticed it first on. lithium-fabric-mc1.17.1-0.7.4
I checked out 7fc9674 locally to verify.

Expected Behavior

The BlockStates is the ChunkSection should be sized according to the Palette size. For example, if the Palette has 16 entries, the BlockStates should be 256. This constraint is enforced by Vanilla.

Actual Behavior

The BlockStates in ChunkSection can be sized differently - for example, I ran into data that was sized for 5 bits (342 long values); however the Palette was 16 entries.

Reproduction Steps

I created a world and flew around for a while. I had set a breakpoint to fire if the above condition was true (BlockStates size = 342 and Palette = 16)

Other Information

No other mods were used.

While debugging I had noticed the paletteSize field was set to 5, which is why the storage was being written with 5 bits. It appears Lithium's optimisation ignores the field and assumes the paletteSize from the number of entries in the palette - and so it just directly copies from the current storage. Why the palette desyncs like that is beyond me - it's far too late at night for me to continue debugging this. But, an obvious fix is for Lithium to also just check that field.

This does not occur on Vanilla because the PackedIntegerArray written is always built sized according to the palette.

While this does not cause any problems on 1.17.1, or even earlier versions, the newer deserialization code in 1.18 snapshots will not accept this data and the chunk will fail to load. While I have already reported a palette deserialization related problem to Mojang already (https://bugs.mojang.com/browse/MC-238056), it does not cover something like this - there's no guarantee they make their deserializer more liberal in this area.

This can be caused by the onResize method recalculating the palette from the blocks in the chunk. If a block becomes absent from the chunk, the new palette will lose 1 block and become size 16.

The problem still exists for chunks written before this fix. Some chunks are affected by this issue and cannot be loaded in 1.18 unless they are first saved with a lithium version >= 0.7.5 .
There will be many users who cannot visit all affected chunks with lithium >= 0.7.5 before upgrading to 1.18.

If mojang does not change the 1.18 chunk deserialization code to be compatible with all chunks written by lithium <= 0.7.4 again, we will provide a method/tool to fix worlds for 1.18.

I think there is a chance Mojang will fix it on their side, as a change in the format is usually guaranteed to not cause world upgrade problems. Mojang usually uses the DFU system to achieve compatibility. I will open a bug report to mojang and suggest that they add a datafixers for this format change.