java.lang.IllegalStateException: TickNextTick list out of synch
yellowcooln opened this issue · 28 comments
The mod was working just fine until last night where the server crashed 15 times in 1 hour with the same error.
`---- Minecraft Crash Report ----
WARNING: coremods are present:
EnderCorePlugin (EnderCore-1.12.2-0.5.45.jar)
Contact their authors BEFORE contacting forge
// Daisy, daisy...
Time: 3/19/19 1:41 AM
Description: Exception ticking world
java.lang.IllegalStateException: TickNextTick list out of synch
at net.minecraft.world.WorldServer.func_72955_a(WorldServer.java:691)
at net.minecraft.world.WorldServer.func_72835_b(WorldServer.java:223)
at net.minecraft.server.MinecraftServer.func_71190_q(MinecraftServer.java:756)
at net.minecraft.server.dedicated.DedicatedServer.func_71190_q(DedicatedServer.java:397)
at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:668)
at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:526)
at java.lang.Thread.run(Unknown Source)
A detailed walkthrough of the error, its code path and all known details is as follows:
-- Head --
Thread: Server thread
Stacktrace:
at net.minecraft.world.WorldServer.func_72955_a(WorldServer.java:691)
at net.minecraft.world.WorldServer.func_72835_b(WorldServer.java:223)
-- Affected level --
Details:
Level name: world
All players: 0 total; []
Chunk stats: ServerChunkCache: 326 Drop: 0
Level seed: 7378376108783325147
Level generator: ID 00 - default, ver 1. Features enabled: true
Level generator options:
Level spawn location: World: (128,64,108), Chunk: (at 0,4,12 in 8,6; contains blocks 128,0,96 to 143,255,111), Region: (0,0; contains chunks 0,0 to 31,31, blocks 0,0,0 to 511,255,511)
Level time: 12548870 game time, 4536354 day time
Level dimension: 0
Level storage version: 0x04ABD - Anvil
Level weather: Rain time: 125356 (now: false), thunder time: 93616 (now: false)
Level game mode: Game mode: survival (ID 0). Hardcore: false. Cheats: false
Stacktrace:
at net.minecraft.server.MinecraftServer.func_71190_q(MinecraftServer.java:756)
at net.minecraft.server.dedicated.DedicatedServer.func_71190_q(DedicatedServer.java:397)
at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:668)
at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:526)
at java.lang.Thread.run(Unknown Source)
-- System Details --
Details:
Minecraft Version: 1.12.2
Operating System: Windows Server 2012 R2 (amd64) version 6.3
Java Version: 1.8.0_201, Oracle Corporation
Java VM Version: Java HotSpot(TM) 64-Bit Server VM (mixed mode), Oracle Corporation
Memory: 819803744 bytes (781 MB) / 3231711232 bytes (3082 MB) up to 8588361728 bytes (8190 MB)
JVM Flags: 1 total; -Xmx9214M
IntCache: cache: 0, tcache: 0, allocated: 12, tallocated: 94
FML: MCP 9.42 Powered by Forge 14.23.5.2768 37 mods loaded, 37 mods active
States: 'U' = Unloaded 'L' = Loaded 'C' = Constructed 'H' = Pre-initialized 'I' = Initialized 'J' = Post-initialized 'A' = Available 'D' = Disabled 'E' = Errored
| State | ID | Version | Source | Signature |
|:--------- |:--------------------------------- |:------------------------ |:--------------------------------------------- |:---------------------------------------- |
| UCHIJAAAA | minecraft | 1.12.2 | minecraft.jar | None |
| UCHIJAAAA | mcp | 9.42 | minecraft.jar | None |
| UCHIJAAAA | FML | 8.0.99.99 | ForgeMod.jar | e3c3d50c7c986df74c645c0ac54639741c90a557 |
| UCHIJAAAA | forge | 14.23.5.2768 | ForgeMod.jar | e3c3d50c7c986df74c645c0ac54639741c90a557 |
| UCHIJAAAA | orbis_api | 0.1.0 | orbis_api-0.1.0.jar | db341c083b1b8ce9160a769b569ef6737b3f4cdf |
| UCHIJAAAA | aether | 0.1.1 | aether_ii-0.1.1-universal.jar | db341c083b1b8ce9160a769b569ef6737b3f4cdf |
| UCHIJAAAA | appliedenergistics2 | rv6-stable-6 | appliedenergistics2-rv6-stable-6.jar | dfa4d3ac143316c6f32aa1a1beda1e34d42132e5 |
| UCHIJAAAA | baubles | 1.5.2 | Baubles-1.12-1.5.2.jar | None |
| UCHIJAAAA | biomesoplenty | 7.0.1.2419 | BiomesOPlenty-1.12.2-7.0.1.2419-universal.jar | None |
| UCHIJAAAA | customspawner | 3.11.4 | CustomMobSpawner-3.11.4.jar | None |
| UCHIJAAAA | endercore | 1.12.2-0.5.45 | EnderCore-1.12.2-0.5.45.jar | None |
| UCHIJAAAA | jei | 4.15.0.268 | jei_1.12.2-4.15.0.268.jar | None |
| UCHIJAAAA | thaumcraft | 6.1.BETA26 | Thaumcraft-1.12.2-6.1.BETA26 (1).jar | None |
| UCHIJAAAA | enderio | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | enderiointegrationtic | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | enderiobase | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | enderioconduits | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | enderioconduitsappliedenergistics | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | enderioconduitsopencomputers | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | enderioconduitsrefinedstorage | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | enderiointegrationforestry | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | mantle | 1.12-1.3.3.42 | Mantle-1.12-1.3.3.42.jar | None |
| UCHIJAAAA | twilightforest | 3.8.689 | twilightforest-1.12.2-3.8.689-universal.jar | None |
| UCHIJAAAA | tconstruct | 1.12.2-2.12.0.115 | TConstruct-1.12.2-2.12.0.115.jar | None |
| UCHIJAAAA | enderiointegrationticlate | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | enderiomachines | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | enderiopowertools | 5.0.40 | EnderIO-1.12.2-5.0.40.jar | None |
| UCHIJAAAA | worldedit | 6.1.8 | worldedit-forge-mc1.12-6.1.8-dist.jar | None |
| UCHIJAAAA | forgeessentials | 12.3 | forgeessentials-1.12.2-12.3.56-server.jar | None |
| UCHIJAAAA | metallurgy | 0.1.1 | Metallurgy-4-Reforged-0.1.1.jar | None |
| UCHIJAAAA | morpheus | 1.12-3.3.2 | Morpheus-1.12-3.3.2.jar | None |
| UCHIJAAAA | mystcraft | 0.13.6.00 | mystcraft-1.12.2-0.13.6.00.jar | None |
| UCHIJAAAA | netherportalfix | 5.3.17 | NetherPortalFix_1.12.1-5.3.17.jar | None |
| UCHIJAAAA | harvestcraft | 1.12.2zb | PamsHarvestCraft1.12.2zb.jar | None |
| UCHIJAAAA | plustic | 7.0.5.2 | plustic-7.0.5.2.jar | None |
| UCHIJAAAA | additionalstructures | 2.2.2 | Rexs-Additional-Structures-1.12.x-v.2.2.2.jar | None |
| UCHIJAAAA | tinkertoolleveling | 1.12.2-1.1.0.DEV.b23e769 | TinkerToolLeveling-1.12.2-1.1.0.jar | None |
Loaded coremods (and transformers):
EnderCorePlugin (EnderCore-1.12.2-0.5.45.jar)
com.enderio.core.common.transform.EnderCoreTransformer
com.enderio.core.common.transform.SimpleMixinPatcher
AE2 Version: stable rv6-stable-6 for Forge 14.23.5.2768
Pulsar/tconstruct loaded Pulses:
- TinkerCommons (Enabled/Forced)
- TinkerWorld (Enabled/Not Forced)
- TinkerTools (Enabled/Not Forced)
- TinkerHarvestTools (Enabled/Forced)
- TinkerMeleeWeapons (Enabled/Forced)
- TinkerRangedWeapons (Enabled/Forced)
- TinkerModifiers (Enabled/Forced)
- TinkerSmeltery (Enabled/Not Forced)
- TinkerGadgets (Enabled/Not Forced)
- TinkerOredict (Enabled/Forced)
- TinkerIntegration (Enabled/Forced)
- TinkerFluids (Enabled/Forced)
- TinkerMaterials (Enabled/Forced)
- TinkerModelRegister (Enabled/Forced)
ForgeEssentials: Running ForgeEssentials 12.3.56 (fccdc2d). Modules loaded: Afterlife, Backups, Chat, Commands, Economy, JScripting, Multiworld, Permissions, PlayerLogger, Protection, Remote, SignTools, Teleport, Tickets, WEIntegrationTools, WorldBorder, factions, mapper
Ender IO: No known problems detected.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!You are looking at the diagnostics information, not at the crash. !!!
!!!Scroll up until you see the line with '---- Minecraft Crash Report ----'!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
AE2 Integration: IC2:OFF, RC:OFF, MFR:OFF, Waila:OFF, Mekanism:OFF, OpenComputers:OFF, THE_ONE_PROBE:OFF, TESLA:OFF, CRAFTTWEAKER:OFF
Profiler Position: N/A (disabled)
Player Count: 0 / 10; []
Is Modded: Definitely; Server brand changed to 'fml,forge'
Type: Dedicated Server (map_server.txt)
The pendingTickListEntries is nothing we touch at all (and it doesn't take a coremod to mess it up).
The pendingTickListEntries is nothing we touch at all (and it doesn't take a coremod to mess it up).
Any idea whats messing it up then?
Nope. I had a quick look over the mod list, but nothing I recognize there looks like it needs to do it.
Nope. I had a quick look over the mod list, but nothing I recognize there looks like it needs to do it.
Alright thank you. I'll try a new forge version and see if that fixes it.
People on my server are saying it only crashes when they work with enderio blocks and machines. But i can't recreate the issue in singleplayer.
Someone else reported a similar bug to IE today, see BluSunrize/ImmersiveEngineering#3423. My current suspect for that report is Project:Red Relocation, since they are the only ones directly modifying those fields that I found. The OP here doesn't seem to have that installed, so I'll probably have to look for a new suspect.
I still have no idea what is wrong. I generated a new world and the error stopped. I don't know what happened to the world but something did. I can upload it if anyone wants to see it.
@yellowcooln Thanks for posting the world file, that helped a lot with debugging!
@ Any devs that read this: After a few hours of debugging and about 5 different versions of my coremod I think I've figured out what's going on, and it isn't nice. Some of the links in the description go to the private Sponge Snowman repo, if you (whoever reads this) don't have access you should be able to find the relevant code in any decompiled version of MC 1.12:
NextTickListEntry implements hashCode and equals in a sensible manner, marking two entries equal if they refer to the same block at the same positions. compareTo uses a different notion of equality: Two entries are equal if and only if a few fields relating to scheduling priorities are equal. WorldServer and especially the fields pendingTickListEntriesHash/TreeSet is/are built on the assumption that a HashSet will use only hashCode and equals to determine equality while TreeMap uses compareTo (and maybe equals?): All insertions (example) into the sets are done only if the hash set does not already contain the new entry. This is probably to prevent multiple updates from being done for the same block (as they wouldn't be considered equal in the tree map).
However if many entries with the same hash are stored in a HashMap (and therefore when this happens in a HashSet) it will use Comparable to "break ties". This means storing the buckets (roughly) as tree sets rather than lists, which are ordered by compareTo. When contains/remove/etc is now called on that map/set it will try to find an element that is in the same bucket and equal according to compareTo (more precisely: in the path from the root of the tree to an equal element). This isn't documented anywhere, the Comparable interface even mentions that compareTo does not necessarily use the same notion of equality as equals (and HashMap/Set doesn't mention the assumption that they are the same).
I haven't looked into the "full history" of the elements involved, but directly before the crash the HashSet#remove-call in the method processing the entries fails even though pendingTickListEntriesHashSet contains 2 (!) elements that should be equal to nextticklistentry.
I've written a small demo of the problem here: The output should be true since the set contains an element with valEqual==0, but the actual output is false.
I'm not sure why this is popping up now (one report here and then a few in other repos just a few days apart), but this would probably be best fixed as a Forge PR and possibly a bug report to Mojang.
I'd need to read the java specs, but that sounds like a bug in HashSet to me, what do you think?
Update:
The javadoc is quite clear here::
boolean java.util.HashSet.contains(java.lang.Object o)
Returns true if this set contains the specified element. More formally, returns true if and only if this set contains an element e such that (o==null ? e==null : o.equals(e)).
"if and only if"
Update: Here's the source of the bug: http://openjdk.java.net/jeps/180
The Java devs seem to consider this to be intended behavior, apart from not being properly documented: https://bugs.openjdk.java.net/browse/JDK-8140741
I've uploaded a small coremod that could help with debugging this issue here. It will print a stacktrace every time one of the methods modifying the tick lists is called from a thread other than the server thread. It's based on the IE coremod (because that's a coremod I have a well-working workspace for), that's why it's called ImmersiveEngineering-...-core, but it works just fine without IE installed. I pushed the code to this branch of my IE fork.
@yellowcooln Could you try temporarily installing that mod on the server (make a backup first, even though there shouldn't be any issues)? There's a good chance it will produce a bunch of log spam, if that happens please upload some of it to https://gist.github.com or a similar site and post a link here. If it doesn't produce any log spam uploading the world would probably be the easiest for further debugging. Is the pack you're using available for download somewhere (e.g. Curseforge)?
@HenryLoenwind I hope you don't have any objections to me posting this here, if you do please just remove this comment. I tried to ask you via DM on Discord, but you have DMs disabled (like most people...).
I've installed the core mod and it crashed when I joined. I thought it would just be a good idea to upload all the files to run the server. Here is a link to the server download. http://sharex.yellowcooln.com/BrokenServer.rar.
@malte0811 If you like you can message me on Discord. Yellowcooln#0001
As a temporary hack, would it be a valid fix to just inject if (this.equals(other)) return 0; into the compareTo method of NextTickListEntry?
I'm reasonably sure that would fix this issue, but would it break the logic that uses the comparability?
NextTickListEntry objects are only put into Set collections, so I don't think it's expected for there to ever be duplicates. So adding this check to compareTo should be a no-op (except for this odd stdlib behavior).
The Comparable is used in the pendingTickListEntriesTreeSet to fetch the elements in order of their scheduled time. Yeah, looks like an identity check won't do any harm---all comparisons there are for non-equal objects. Only a very tiny performance hit due to the many comparisons in equals().
Make it so, number 1098! :-)=
I am having the same issue https://pastebin.com/raw/e0rxJmUK
Yes, because compareTo with the equals-check added is no longer transitive. I really should have noticed this issue 3 months ago. This breaks contains-checks of any sort, for both of the sets. If 1 and 3 are equal (according to equals), a situation like this can easily happen (either in a single bucket of the HashSet or in the TreeSet in general):
2
/ \
1 4
Checking whether the set contains 3 would return false since 3>2 and 3!=4, even though it contains an element that is equals to 3 according to both (the modified) compareTo and equals (1). The demonstration I posted above also shows this, the output remains the same when adding if (equals(o)) return 0; to compareTo.
I've implemented the compareTo fix in RandomPatches, yet the issue is still occurring. I'm almost certain the patch is being applied correctly as I've tested with different priorities and scheduled times. Any ideas?
Thanks. In that case, I will attempt another fix by replacing all HashSet#contains and HashSet#remove calls with my own.
EDIT: Done.
According to @runescapejon, the fix appears to be working.
Wouldn't it be more efficient to replace the HashSet<T> with a custom "PureHashSet<T>" which would be a wrapper around a HashSet<EqualsWrapper<T>> (where EqualsWrapper<T> stores a T and only implements equals and hashCode)? remove and contains would be (close to) constant time, rather than linear as they are now. That would also require less bytecode modification, only the initializer would need to be replaced.
Done. Do you see any issues?
@malte0811 I am having the same issue with my server and I have 0 knowledge in scripting. Could you please help me? My discord is 911HoldPls#8141.
This seems to have reemerged in 1.16.1 with Forge 32.0.75. Logs are very brief. I am willing to help debug. The crash occurs even with the fix provided by RandomLab's Patches mod. Very irregular but persistent, seems to increase in frequency the longer a world is played (not in one session) and the more mobs are nearby the player.
Any data you need from me I will provide.
EDIT: is that coremod functional with 1.16?