Applied Energistics 2

Applied Energistics 2

156M Downloads

Meteorites from AE2 will cause server to hang forever if it hits certain blocks from BYG [1.16.4 Forge]

TelepathicGrunt opened this issue ยท 8 comments

commented

After several hours of debugging, I narrowed down a server hang to the meteorite from AE2 but only in certain circumstances.
BYG: https://www.curseforge.com/minecraft/mc-mods/oh-the-biomes-youll-go
World Blender: https://www.curseforge.com/minecraft/mc-mods/worldblender
World Blender config: world_blender-blending.zip
World's seed: 1
Entered the dimension by doing /execute in world_blender:world_blender run tp ~ ~90 ~ and heading to world origin and exploring.

The hang is basically, chunks in world blender's dimension stop generating, mobs stop moving, block updates stop, lighting threads die and spam log, and stays stuck forever. This happens constantly and was hard to figure out which mods it was until I started taking out mods verrrry careful and over hours. In the end, when I remove either AE2 or BYG, the hang never happens. Here's the snapshots which can be opened in VisualVM or Eclipse to read VisualVM's profiling of the threads. Notice how in both snapshots, it doesn't say exactly what is causing the hang. Especially in the first one. The server thread just waits on the chunk and stays waiting.... forever. Even the logs are useless. The attached log shows me teleporting to several meteorites in Overworld but the moment I teleport into World Blender's dimension, I hang.
snapshots.zip
latest.log

If I turn off the meteorite in AE2's config, the hang never occurs which is how I know this structure is the cause. Something about it is forcing the server to hang forever but only when it hits something from byg. Probably some sort of feature or mob or something that isn't present in the Overworld as World Blender also merges everything from the nether and end into the dimension. This is as much info I was able to gather but hopefully this helps

commented

For anyone else reading this, here's how to manipulate the configs to prevent the meteorite from spawning in world blender's dimension and causing the hang when byg is also on.

In Applied Engergistics 2's config, do this:
meteoriteBiomeBlacklist = ["world_blender:general_blended", "world_blender:cold_hills_blended", "world_blender:mountainous_blended", "world_blender:ocean_blended", "world_blender:frozen_ocean_blended"]

in world blender's blending config, do this:
blacklistedStructures = "appliedenergistics2:meteorite"

The hang should no longer occur while keeping the meteorites in other dimensions

commented

Wait, BYG is on Forge now?

ps: Try turning on AE2 debug-logging in the config, because sometimes we'll swallow exceptions and don't log unless that's enabled

pps: These look liker profiler snapshots, but those don't really help. You need a thread-dump essentially.

The only really unorthodox thing that AE2 is doing in it's meteorite generation is creation of the chests containing the presses since it currently does not use a loot-table (it's trying to globally pre-distribute at least one of each press before making it random).

Other than that, I don't really know how our code would cause this, since we're essentially doing bog-standard structure-piece generation. We do need to sample the noise maps for height, but that I believe should not cause a chunkload.

commented

Oh the Biomes You'll Go has always been on forge. The Fabric port was done later.

2 Thread dumps:
https://hastebin.com/ipubununab.properties
https://hastebin.com/enagiwewub.properties

The dump doesn't show why the meteorite is killing the world and the main server hangs in different places each time I run it. The log with debug config on is also unhelpful: https://hastebin.com/buqimerani.sql

To triple check, I updated all mods including AE2 (person had an older version before the dimension whitelist config) and then I made sure meteorites are whitelisted only for overworld and world blender blacklists the meteorite from being imported. Once the meteorite doesn't spawn in the dimension, the seed 1 world does not hang at all. I even explored a massive amount of the world still no hang. This is reproducible every time.

It's definitely something with the meteorite. I don't know what. I did place a ton of sky stone block and chests with fill command and no hang. Only when the meteorite is generating does it hang.

commented

edit: No, it's not "definitely something with the meteorite". Worldgen is so stupidly bad that we've had many cases in which a meteorite simply triggered the fucked up design of some other mod's worldgen, leading to crashes down the line.

Oh. This was already visible in the debug-log you pasted above.

All your world-gen threads are killing themselves when they apply the lighting, making the server wait indefinitely for the chunks to generate:

[17Dec2020 16:41:49.208] [Worker-Main-27/WARN] [net.minecraft.util.Util/]: Worker-Main-27 died
java.lang.NullPointerException: null
	at net.minecraft.world.lighting.SectionLightStorage.func_215521_h(SourceFile:96) ~[?:?]
	at net.minecraft.world.lighting.LightEngine.func_215471_c(LightEngine.java:129) ~[?:?]
	at net.minecraft.world.lighting.LevelBasedGraph.func_215469_a(LevelBasedGraph.java:122) ~[?:?]
	at net.minecraft.world.lighting.BlockLightEngine.func_215623_a(BlockLightEngine.java:118) ~[?:?]
	at net.minecraft.world.lighting.WorldLightManager.func_215573_a(SourceFile:41) ~[?:?]
	at net.minecraft.world.server.ServerWorldLightManager.func_215579_a(SourceFile:135) ~[?:?]
	at net.minecraft.world.server.ServerWorldLightManager$$Lambda$6222/1513905213.accept(Unknown Source) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_51]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[?:1.8.0_51]
	at net.minecraft.world.server.ServerWorldLightManager.func_223117_a(SourceFile:135) ~[?:?]
	at net.minecraft.world.server.ServerWorldLightManager$$Lambda$6217/1006753112.run(Unknown Source) ~[?:?]
	at net.minecraft.world.server.ServerWorldLightManager.func_215603_b(SourceFile:170) ~[?:?]
	at net.minecraft.world.server.ServerWorldLightManager.func_223119_a(SourceFile:108) ~[?:?]
	at net.minecraft.world.server.ServerWorldLightManager$$Lambda$6209/1394231676.run(Unknown Source) ~[?:?]
	at net.minecraft.world.chunk.ChunkTaskPriorityQueueSorter.func_219083_b(SourceFile:58) ~[?:?]
	at net.minecraft.world.chunk.ChunkTaskPriorityQueueSorter$$Lambda$6153/194242279.run(Unknown Source) ~[?:?]
	at net.minecraft.util.concurrent.DelegatedTaskExecutor.func_213148_e(SourceFile:94) ~[?:?]
	at net.minecraft.util.concurrent.DelegatedTaskExecutor.func_213145_a(SourceFile:137) ~[?:?]
	at net.minecraft.util.concurrent.DelegatedTaskExecutor.run(SourceFile:105) ~[?:?]
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) ~[?:1.8.0_51]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_51]
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_51]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1689) ~[?:1.8.0_51]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) [?:1.8.0_51]
commented

I would more or less assume the same. Something has an effect on the lighting, which then kills the world gen threads and cause it to stall.

Also I would guess that meteors are not the actual bug, just merely triggering it. They destroy blocks, which could case lightung updates. Then mixing every biome into the same world can easily lead to a various bugs. Like nether biomes expecting a bedrock ceiling with no sky light at all and once something tries to update it, it will crash or similar things.

I am currently not aware of any mod adding structures similar to meteors, but I would really expect that if some does it, it could cause the same issues.

commented

While I do find that lighting threads tend to die when servers hang, you are right that the lighting thread is dying too quickly in this case for it to be simply a case of it being a side effect of server stalling.

However, I have tested World Blender extensively with many many other mods and while I do find strange issues, this lighting thread killing has never been seen before. It is specific to AE2's meteorite.

In fact, I also would like to refute @yueh's claim that it is just a normal occurrence of structures that carve or nether biomes (world blender uses it's own biomes. No nether biome spawns in the dimension). Again, no other mod has the same issue and I even went ahead and created a datapack that turns bastions and outposts into an insanely massive carving structure. Those massive carving structures does not cause the threads to die or the server to hang so it seems Jigsaw structures can carve safely.

Datapack: carving_structure.zip

image

I did manage to reproduce this issue using the Fabric version of all 3 mods with the same world so it isn't a Forge issue. However, adding Phosphor on the Fabric version does prevent the hang so whatever the meteorites are doing, it's unsafe for vanilla lighting and Forge doesn't have phosphor that they can use to prevent the hang.

It is possible that this issue does occur in the overworld but much more difficult to trigger. It honestly might be a good time to revamp the meteorite code so that it doesn't cause as many floating leaves in overworld and might end up fixing this lighting issue as well. It might be possible to recreate the meteorite using jigsaw structures with custom processors that runs custom code for the fluid filling, chest, crater block replacing, etc. Just a suggestion.
image

Yeah, I am just as stumped as you and this as much info I can gather.

commented

Meteor gen happens way later than other structures or carving, so there might be a difference in terms of how lighting is applied. But thanks to forge being really broken currently, it's pretty impossible to look at the actual process without decompiling minecraft by hand.

Further the support for custom structures in forge is still completely missing, so anyone has to come up with their own hack to make them work. And I wouldn't be surprised, if there is something else breaking it in some way. As far as I can see it looks like BYG does not have a maven, so getting it running indev is a mess and I am honestly now annoyed enough by forge for making it harder and harder to debug cross mod issues every update, that for now I do not really care anymore when it needs more than adding a temporary maven dep and maybe repo to the build. In 1.7 you could just drop a jar into the mods folder and it would work, that stopped during 1.10 or 1.12 but at least it was still possible to just add them to the classpath manually. Now not even that works anymore and if it is not added as part of the whole build proces without regenerating everything, it does not work at all anymore.

In terms of the foliage, feel free to submit a PR to fix it. As far as I can tell, this is still an artifact on how minecraft handles spawning trees&co at an offset to the actual chunk grid or at least spawning a tree will modify the adjacent chunks after they are generated. So the only option would probably to regenerate the whole meteor each time an adjacent chunk is completely generated. Which more or less means generating each chunk 10 times. With the chunk gen already being slow as heck, I really would try and avoid putting anything more into it. So if you have a better idea while keeping the current flexibility and randomness, I would be really interested in seeing another working approach.

commented

@TelepathicGrunt I figured this one out. Vanilla bug :-|

It happens when the meteorite clears an entire chunk section that previously contained a light-emitting block.
net.minecraft.world.chunk.ChunkPrimer#lightPositions is only added to, never removed from.
But the code that iterates over lightPositions and tries to propagate emissive blocks assumes that the chunk section the block is in is still populated (which, after the meteorite clearing it, it isn't).

So this is probably more prevalent with BYG because it had spawned light-emitting blocks above the meteorite.

edit: Workaround mixin in #4935.