If a player uses a bunch of bit blocks in their build on a server, they will eventually cause the players loading the chunk(s) to essentially get chunk banned. Players inside the render distance of the bit blocks will not be affected until they leave and re-enter. Spreading the blocks around seems to alleviate the issue, and it's only really caused when there's a lot of them in a single area. Unfortunately, that single area is too small for some of the more creative players.

No error is given in the console, just X lost connection: Disconnected. The player receives a typical disconnection message, and the game itself doesn't crash.

Twice now I've had to reset the affected chunks. One build was a massive mural that was a little over halfway finished until issues arised. The second incident happened last night and that one was a simple house with some decorations added in.

Here's roughly how many bit blocks you can have in a single area before it starts causing disconnections.

There is way more blocks inside the house. Some of the walls are bit blocks as well.

And here's the wall without the mural and with the mural a day before the disconnections happened.

Mods Pastebin

At first I thought it was some mod conflict and I did find some bad mods which I removed, but that didn't fix it. I also tried removing every mod other than Bits And Chisels and that didn't help.

It would seem that XL Packets Fabric doesn't actually function properly in 1.16.5?

I have identified krypton-0.1.2 as the culprit. For some reason, when it's installed on the client it causes people to disconnect when loading a lot of Bits blocks. The mod itself still works on the server-side. Removing Krypton solved the issue.