I tried generating a world fill with a server instance of 8GB of memory and it seemed OK until it ran out of memory.. and then just sat there. No memory was ever freed and the generation could not continue. So I "wb fill cancel" and "stop" to shutdown the server. But then it got stuck on "Saving chunks for level 'world'/Overworld".

So I tried again but this time with 32GB of memory but this had the same problem. Memory usage just ticked up until there was no more memory available. Again, I canceled the fill, performed a "save-all" which completed instantly, and tried to stop the server but it hung on saving chunks again.

First of all, where is the memory going? And who isn't freeing it? I have ONLY WorldBorder and LagMeter loaded with no other plugins. Secondly, why is it hanging when the server stops? Are chunks not being written to disk by WorldBorder? Why did save-all work OK but the server still had chunks to save?

While I waited for the server to stop by itself, I examined the world's region folder. No new files were being written but the existing files were changing. But it wasn't just each file being touched once; it was often the SAME files being touched again and again. For example, I could filter by modification date, click on a file and watch it scroll away, and then watch it scroll past again as it is repeatedly modified.

Here is a screenshot of what I am describing:

https://www.youtube.com/watch?v=U0PRyGh9o2o

Is this a WorldBorder problem or a bug with Spigot's chunk generation?

Used a command block method to generate chunks with a 8GB instance and memory usage was fine. This seems to be a WorldBorder-specific issue:

https://www.reddit.com/r/Minecraft/comments/3r7xtm/getting_blocklag_in_19_get_rid_of_it_pregenerate/

This issue is likely related:
#75

@Brettflan:

This issue is likely related:
#75

I used a brand new server to generate chunks and the only plugins were WorldBorder and LagMeter. There were no commands or other plugins running.

As mentioned previously, I ended up using the command block method to generate a 20000x20000 world which took about 24 hours; it was VERY slow. I calculated that it would take about 8 days to double that. And double that double take about 72 days.

This large world was done on an 8GB instance and had no problem freeing memory or shutting down the server. It was also performed on the same version of Spigot as WorldBorder was running on (at the time).

I have no had a chance to test with the latest spigot yet.

I think that the memory exhaustion is due to the set of chunks waiting to be unloaded growing indefinitely (as long as the server is at or near the point of being overloaded) after this recent Spigot change. A fix has gone in; @zhro, is it better with a fresh Spigot build?

it seemed OK until it ran out of memory.. and then just sat there. No memory was ever freed and the generation could not continue

Once WorldBorder has detected the lack of memory and has paused itself (saving the server from blowing up with an OOME!), the unloadable chunk queue will continue to drain (and faster, if the server now has time to spare). This will free up memory and so WorldBorder will unpause the fill operation, if the JVM decides to do enough GCing. In my testing (admittedly, with a much smaller radius and heap size), it doesn't and so WorldBorder never knows that it's safe to continue. When I did force GC on the server after the queue had drained for a while, WorldBorder would immediately restart the fill.

I canceled the fill, performed a "save-all" which completed instantly, and tried to stop the server but it hung on saving chunks again.

I'm not certain why this happened. My guess is that it's because, at shutdown, the server can sometimes pause between writing chunks back to their region files (vs. /save-all flush, which writes chunks back as quickly as it can).

While I waited for the server to stop by itself, I examined the world's region folder. No new files were being written but the existing files were changing. But it wasn't just each file being touched once; it was often the SAME files being touched again and again.

That's normal. The server writes chunks back to their region files in no particular order.

I confirm this issues as well.

I do ask that you read through issue #75 linked above. I'll just reiterate a couple of important points from there anyway since they're not answered:
What Fill frequency are you using, the default, or have you specified a higher frequency? If you've specified a higher than default frequency, try using the default.
Otherwise, have you tried setting Fill's autosave to a much higher value (maybe try 300 seconds or so) or even disabling it to see if that makes a difference (command "/wb fillautosave [seconds]", set 0 to disable)?

I've been getting similar issues on a dedicated CentOS7 server I recently purchased. Spigot 1.12.2.

It just keeps going and going until it eventually runs out of memory, getting down to loading 1 chunk per update before crashing. It never releases memory.

I've tried 1000, 300, and the default frequency. I've also, after googling after the issue, tried raising the fillautosave value to 300 (every five minutes), to see if it'd result in different behavior. It does not.

Additionally, I've confirmed despite the constant information stating that the chunks are being saved to the disk, if the fill is cancelled at some point (say several hours afterwards when it begins grinding to a halt), and then restarted after the server restarts and has saved the overworld and it's chunks (takes a very long time to do so), restarting the fill command results in it seemingly missing a very large amount of progress, going back by several tens of percentages, and falling back into the same slope of slowly grinding down - which further seems to confirm that it is not writing the data and releasing it from memory.

The Fill will include in the total number of processed chunks any chunks that have neighboring chunks existing on all 4 sides (thus being fully generated with all structures and such), but will skip right past them without loading them since it can see there's no need. So while it still counts them as processed, they take almost no time at all to skim through. Of course any chunks on the outside edge of what's generated will all end up needing to be loaded again to ensure full generation. You should notice it quickly glide through to a percentage not far below where it was at before you restarted the server.

Have you tried temporarily disabling world autosave in the Bukkit config, assuming that setting still exists in Spigot? If you can do that, you might change the Fill autosave back to something like 30-60 seconds. There have been reports of the two world autosaves conflicting for some people.