When the layout block TE is regarded as no longer in view, then the rendering will stop. Granted you have to be close to it to place the blocks in the first place and it is only at certain angles.

Look to see if this is simply a bounding box issue or the global renderer fix is needed

Adding an infinite AxisAlignedBB solved the issue, so I need to calculate the correct bounding box to return.