High memory usage in StructureTemplates
Ricket opened this issue · 9 comments
Context Environment
Minecraft version: 1.12.2
Ancient Warfare 2 version: 2.7.0.1032
Minecraft Forge version: 14.23.5.2854
If you are using a modpack, link to the pack: MC Eternal 1.4.3.1
Expected Behavior
Not consuming a whopping 536mb of memory
Current Behavior
I took a heap dump to inspect a memory leak (unrelated) but then noticed that the next largest consumer of memory aside from the leak is this mod, using a lot of memory for StructureTemplates.
The templateData array is very large and I wonder if it can be unloaded, compressed, or otherwise reduced somehow...
Steps to Reproduce
- Start MC eternal, log into a game
- 536mb of memory is taken up from WorldGenStructureManager
Possible Solution
I have not used this mod yet so I don't know what these structures are but as mentioned above, maybe these large arrays can be unloaded when not needed, or compressed since it appears they are sparse (mostly 0s), or some other solution to save memory.
Well this really comes down to compromise between performance and memory usage. If these were not loaded they would need to be loaded into memory as soon as a structure would need to generate in world and I am sure that IO operations at that point would definitely have a great impact on performance. The same goes for compacting these arrays. all of those 0s are actually needed as they point to the block in a map of blocks used in a structure. the compacting would mean uncompacting a the time the structure is being generated in the world which would easily for these huge structures add seconds to the time it takes for them to build.
But at the same time they are only needed in memory if you actually use them in the game - they are generated or you spawn them using creative tools. And they are also only needed on the side that actually generates these structures so in multiplayer there's no need for them to live in the memory of clients. You can turn them off in structure config if not needed.
I may take a look at more efficient way of handling these in the future port to newer versions of MC, but anything that would save memory and have good performance at the same time would require very significant changes. That's why I wouldn't be targeting to do anything in 1.12.2 for this.
they are also only needed on the side that actually generates these structures so in multiplayer there's no need for them to live in the memory of clients
Well that seems like an easy fix then, right? Could you fix the mod to load the data when the server actually starts up or first generates structures (i.e. then clients would never load it), instead of on startup?
Seems like you could pretty much just move this line https://github.com/P3pp3rF1y/AncientWarfare2/blob/1.12.x/src/main/java/net/shadowmage/ancientwarfare/structure/AncientWarfareStructure.java#L168
down inside serverStart https://github.com/P3pp3rF1y/AncientWarfare2/blob/1.12.x/src/main/java/net/shadowmage/ancientwarfare/structure/AncientWarfareStructure.java#L202
maybe?
the issue with easy fix for clients as Dan points out is we don't know know when you're starting minecraft whether you will play on server or in singleplayer and loading structures when singleplayer world loads/generates is a very bad experience because that time will get basically much longer than what players are used to without any explanation as to what is happening.
It's really players that know that they will only play on server who can turn this off by themselves.
Like Pepper said, there is already a config option for clients.
Could you fix the mod to load the data when the server actually starts up or first generates structures
This would have a huge issue during world generation as your world would basically freeze for a minute. Poor user experience and most users would just restart the game thinking it crashed.
Moving it to the beginning of the world loading after MC initialization is also not a good solution, as that would make world loading very slow and it would make every world loading slow if, you close one and reopen another (or the same world), not just the first time.
Well, I wasn't, but I just now joined the discord (didn't realize you had one) and then I think you banned me. So I'm not sure whether you're from the future...
I checked my logs, it took 7 seconds to load structures on startup, not a minute. You're wasting 537mb of everyone's RAM (and 7 seconds for everyone during pack load) just so that the single players can save 7 seconds during loading their world. That's assuming you even need all ~1300 structures in order to load a world. If they're loading into an existing world, I assume you wouldn't even need any of them since they'd already be generated. And if they're making a new world, surely you don't put all 1300 structures in the spawn area.
If the config is your fix then you should default the config to turn off the 537mb. But surely there's a better fix to be had here, where everyone doesn't need 537mb of ram wasted.
Anyway for the sake of my own memory can you tell me which config I should turn off to save this waste? is it enable_world_generation or enable_structure_generation or enable_town_generation?