If there are too many tiles in a chunk it causes the chunk to be to big and throws an error like below when it gets send from the dedicated server to the client.

lost connection: Internal Exception: io.netty.handler.codec.EncoderException: java.lang.IllegalArgumentException: unable to fit 3185468 into 3

That's a known issue. Unfortunately there is no way to fix it. You will have to live with this limit. Keep in mind that everything stored in a chest or other kinds of storage is equally heavy. Means if you put thousands of LT recipe with lots of tiles inside one chest you might exceed the limit already.
Built carefully and spread things (not put too much detail in one chunk). Spheres in a high grid will require a ridiculous amount of tiles, so make sure to avoid these kind of stuff.

It's not that easy ... there is a solution to this problem. You can install CubicChunks, it will completely remove this limit.

Is it not possible to check how big the chunk is and warn the players about it?

But also it will remove any compatibility with vanilla worlds.