Immersive Engineering

Immersive Engineering

134M Downloads

[1.16.1] Server Crash: Ticking entity (Loading connector)

Mowmaster opened this issue ยท 20 comments

commented

Before reporting:

This might be a continuation of the #4111 crash, but i'm not seeing any existing issues for it crashing servers.

Description of the issue:

Server Crashes Every Few Hours, My Player is in the Nether, and the Cords given for the connector erroring should be outside of spawn chunks in the overworld. So im not quite sure why it'd even be loaded, the players setup using the connector has a water wheel setup connected to a LV capacitor and from that this connector which is on top of an IE treated wood pole, is linking it to the metal press.
Crashing connector

Crashlog:

Server Log: https://pastebin.com/jYmdHWN5
Server Generated Crash Report: https://pastebin.com/riv96QmR (There are around 5 similar reports of these now, if you need more)

Versions & Modlist

Server Pack:
https://pastebin.com/9niFAH4i

I dont have another available server do do testing on, so i havent tested in a reduced modpack.

IE Version: ImmersiveEngineering-1.16.1-3.1-112
Forge Version: 1.16.1 - 32.0.108

commented

Unfortunately I'm experiencing this issue too. It's triggered by a player's build--It has never happened while I was online, I can't reproduce it reliably, and the player isn't quite sure what she's doing (she thinks it just happens if she has recently connected after no one was around for some time--loading chunks then?), but it happened several times so far. It's a server-side crash, like the issue author's.

Crash log:

Crash report: https://gist.github.com/Protected/9382d5bf5dad459da11e4e82d5945837
Tail of server log (uncut): https://gist.github.com/Protected/9ee074cc359334d7019cc5345d36d30c
I added the wire net in a more human-friendly format as a comment.

The following screenshots depict the wire net mentioned in the crash:

1
2
3

Versions & Modlist

ImmersiveEngineering-1.16.1-4.0.0-pre.27
Forge: 32.0.108

(Mods listed in the crash report.)

commented

Correction, I managed to reproduce it in my test server just now.

  1. Start the server.
  2. Connect for the first time near this wire: nothing happens.
  3. Disconnect near this wire. Wait a bit (less than one minute is enough).
  4. Reconnect near this wire. server crashes reliably.
commented

The keen-eyed among you may notice from my third screenshot above that there is a second connector on the dynamo. This connector is part of a second linear wire, which I have noticed may also be reported as the cause for the crash on some instances of the crash. Since Mowmaster also had a block with two connectors (the capacitor), I figured, thinking myself very smart, that I would try replacing this setup with a single connector on the dynamo plus an added relay for splitting into the (formerly) two wires:

image

It didn't work; The crash now lists all connectors for both (former) wires as a single net, but it happens in the same circumstances.

commented

Crash still happens after destroying every connector, relay and wire in the vicinity except for the relays {23, 70, 177} and {15, 69, 176}, plus the wire that connects them (all three are shown in screenshot 2).

Crash stopped happening when I destroyed {23, 70, 177}, but started happening again when I placed a new relay on {23, 70, 177} and connected this wire again.

commented

Having noticed {15, 69, 176} was on the corner of the next chunk (adjacent to {23, 70, 177}'s chunk), I tried repositioning this relay to the following locations, reconnecting the wire each time:

  • To the corner of {23, 70, 177}'s chunk
  • To the edge, but not the corner, of {23, 70, 177}'s chunk
  • To a non-edge position of {23, 70, 177}'s chunk

None of these changes prevented the crash from happening. Probably irrelevant, then.

image

commented

I tried destroying the remaining IE blocks in the chunk that were not part of the wire, then moving {23, 70, 177} to the next block horizontally. It did not solve the crash.

Since by now both relays had been moved, I had to surmise the crash either persists if the net isn't completely destroyed on unload, or has nothing to do with the net directly. I destroyed both relays and built a new pair of relays and wire (only two blocks apart) on a different location within the same chunk (several blocks higher up). No dice.

Then I removed both relays and wire from the chunk, leaving it completely free of wires and relays, not to mention any other IE blocks. I built two relays and a wire connecting them on the side of the windmill, which was the next chunk over. Still crashing.

At this point I have to conclude that these chunks are hopelessly corrupted. I destroyed the relays and wire and did not replace them, leaving the area completely free of connectors, relays and wire. Obviously this "resolved" the crash, but this solution will prevent my player from using any IE content in her building (the windmill, dynamo and nearby external heaters do not cause any issues, and probably neither did the coke oven, pump, pipes and blast furnace I destroyed during these experiments; just the wiring).

I shut down the server gracefully, set wires.sanitizeConnections to true in the config (this is still on the test server) and launched it again. It didn't seem to do anything. Placing two relays and connecting them with a wire in the problematic chunk reintroduced the crash.

I'm all out of ideas!

commented

I shut down the server gracefully, set wires.sanitizeConnections to true in the config (this is still on the test server) and launched it again. It didn't seem to do anything.

I wonder; might this be related to #4249 ?

commented

I do want to point out in my crashlog that on line 119 i had a piglin entity with a null Vehicle, it would go through the portal and trigger this crash it seems. also killing this mob would result in a crash, so in my case i set forge to remove erroring entities and so far it seems to have resolved itsself as far as i can tell. (This doesnt solve the IE TE crash portion, but i see someone has done some extensive testing to that end)

commented

Relevant because my modpack has known portal issues (reported in another ticket): This player does not have a portal; there is no portal in range.

commented

I haven't managed to reproduce this issue yet, I suspect there's ghostloading (accidental chunkloading) involved.
@Protected: As you seem to have a reliable setup for this issue it would be great if you could enable enableWireLogger in the IE config, trigger the crash and send me the resulting log. The log size should be within reason as long as you don't turn on validateNets, but if it's above what the usual paste sites can handle (which I don't expect) I'll happily take a compressed log file via some file hosting service as well.

commented

I can just put it right here: latest.log

commented

Today, using NBT explorer, I checked the region file and confirmed that:

  • There is no connector listed among the chunk's tile entities when the crash doesn't take place (all connectors are confirmed to have been destroyed).
  • When a single connector is added to the chunk (no wires) it triggers the crash, and the region file lists only that single connector for the chunk (no duplicates).

image

There are 72 tile entities listed in the chunk, most being parts of the coke oven, blast furnace, a few pipe segments and a pump.

  • Restarting the client after unloading and before triggering the crash doesn't keep the crash from happening.

I peeked at the source code to get a vague notion of how it works. In both cases, a single load event seems to be triggered after I connect, as expected. Since this crash requires a short delay between connections, I expect this is legitimate behavior; the chunk unloaded between visits.

What I'm not seeing that should probably be there is an unload event for this connector. I decided to run the latest test for you again to get a better log, except this time I'm using the single connector, no wires, and immersive portals isn't in the pack just to be safe (gets rid of the mixin in some stack traces).

Here's the result: https://gist.github.com/Protected/e9d3aa2a1311c92a7327541a6b6c9e31

Note how:

  1. I join and the connector loads: LOAD at BlockPos{x=23, y=60, z=177} (tile blusunrize.immersiveengineering.common.blocks.metal.EnergyConnectorTileEntity@bfe5380)
  2. I disconnect. Nothing happens.
  3. More than a minute later, I join and the connector loads: LOAD at BlockPos{x=23, y=60, z=177} (tile blusunrize.immersiveengineering.common.blocks.metal.EnergyConnectorTileEntity@3347bb15). Server goes nuts.
  4. I am disconnected by the error. During my disconnection, the connector unloads: UNLOAD at BlockPos{x=23, y=60, z=177} (tile blusunrize.immersiveengineering.common.blocks.metal.EnergyConnectorTileEntity@bfe5380)

The connector that unloaded during my disconnection was the first instance. Why did it not unload earlier?

Next I went to my own building (which is out of range of this one) and disconnected. My connectors loaded on approach, then unloaded on disconnect. A quick count yields 138 LOADs and more than 150 UNLOADs, however, notably, the other player's connector never UNLOADs. Could this have something to do with distance to spawn point?

commented

Possibly related to #4152 and friends.

commented

Changing the world spawn point to near my building did not affect the situation in both cases beyond what you'd expect (loading stuff on startup).

Changing the world spawn point to the building where the crash can be triggered prevents the issue by keeping the chunks loaded at all times, as you'd expect (it no longer loads the connectors more than once--at startup--so it doesn't matter that they don't unload).

commented

I tried the following things that failed to stop the problem (after spawn point was moved away again):

  • Moving the connector to different altitudes, biomes and chunks in the vicinity
  • Destroyed all IE blocks and multiblocks in the vicinity one by one until none was left left
  • In addition to Immersive Portals, also removed Quark and Engineer's Decor

I looked around the area for oddities, as well as underground all the way down to bedrock, and found nothing.

I failed to reproduce the problem closer to the center of the world but still in the same quadrant.

commented

Done. Hope it doesn't go to spam.

EDIT: Your MTA has rejected my e-mail with a blacklisted IP error. I can't find my IP listed on any public blacklist, so I sent them an unblock request. Waiting to hear from them.

commented

That edit would explain why I couldn't find your mail anywhere. Discord would be an alternative, malte0811#3564.

commented

It would be great if you could send me a world with this issue for testing (either here or by email, address is on my profile). I haven't managed to reproduce this despite a lot of attempts, and I'm not sure what "component" of the issue I might be missing. In my tests loads and unloads always "pair up" in the correct order, and I didn't find any obvious code paths that would cause this out-of-order loading and unloading.

commented

Thank you again for sending me the world files! I've gone from "how can this ever happen" to "why didn't this happen in my tests". If I'm reading the code correctly loading a single chunk (possibly per dimension with wires in it) using /forceload should "fix" the issue, I'm working on a more permanent fix for the next release.

commented

I don't think this is happening anymore after the fix. At least I moved the spawn point away from the problematic location and nothing crashed anymore.