When memory has nearly filled up, it seems chunks are culled to ensure it doesn't fill up (?). This seems to affect nearby chunks first, causing the subchunks I'm in/around to get unloaded.
Updating a single block within a subchunk causes it to be rendered again, as does F3+R/F3+T. However, this is tedious and happens nearly every single time memory fills up.
I am using Quilt + Sodium + Nvidium + Bobby.

Before:

After:

Note the terrain memory usage and memory limit on the right.

ETA:
There is nothing in the logs about this.

That's either an Nvidium thing, or a thing your graphics driver does.
That line is not present with just Sodium, so I'm assuming Nvidium adds it, and it's GPU memory managed by Nvidium (or the graphics driver on behalf of Nvidium).

Bobby only uses regular memory (the second line on the right) which is no where near full (and there's also no mechanism for unloading chunks when it gets full).
I'd recommend you open an issue on Nvidium's issue tracker, I strongly suspect the same issue would happen without Bobby if there was another way to load lots of chunks.