[1.12] 0.3.1 Server Stall Crash
Krutoy242 opened this issue ยท 23 comments
After updating 0.3.0
=> 0.3.1
, loading old world on dedicated server causing extremely low TPS (about 20000 ms for 1 tick) and eventually, crash (tick time treshold).
On 0.3.1
, before i join the world i made Spark profile, because without players server run 2 TPS https://spark.lucko.me/49r1skzRXG
After i join i got crash after few minutes.
Downgrading to 0.3.0
and loading world from pre-update backup fixed issue.
debug.log with crash info
Crash Log
Also
This may be related to Lootr, but in another version of my modpack (on lootr 0.3.0 or even earlier), we had issue with carts x50 times copied on same position. This happen only once, so we didnt make an issue.
It may be unrelated, but this time i tried to call /cofh killall cart
, and then server print 16831
same errors:
[Server thread/WARN] [net.minecraft.world.storage.loot.LootTable]: Tried to over-fill a container
Yes, this is E2E-E server. There was one player that played for a week on this version. But only after week server started to stall
Ok, i played some time again. Still stalls.
As before, removing Lootr fix stall issue.
lootr-1.12.2-0.3.2-BETA-2
@Krutoy242 Hmm... not sure why it would still happen. Is this an E2E-E server you're testing on?
@Krutoy242 Can you give the beta I linked here a try on your server using the backup and let me know if the behavior is any different?
give the beta a try
Sorry, i spent all my free time today on debugging already ๐ฅ
So maybe in week or two.
More info:
The stall and crash even happen on 0.3.0
.
I just ran /cofh killall minecart
and removed about 20,000
Lootr's Minecarts ๐จ
I made profiler with it (1 tick about 30000 ms) https://spark.lucko.me/5BJxm2CRGg
I've tweaked the minecart spawning to try and avoid this duplication; that is the reason for the lag. (This also includes the changes from the beta I linked above.)
Are you running a normal Forge server, or is it Sponge/one of these other hybrids?
normal Forge server
Im running normal server.
tweaked the minecart spawning
How i can test this? The problem happened after a week of play or so.
Do you know how to reproduce the issue?
No, I haven't been able to reproduce it, but there was some logic inherited from Lootr 1.16 that could potentially try to spawn the same minecart more than once if chunks were unloaded/loaded a certain way. I don't think this is necessary in 1.12 so I removed it. My best suggestion would be to simply play for another week and see if the problem is repeated.
@Krutoy242 In the meantime, 0.3.2 has been released on CurseForge; I would expect it to fix your problem unless I've missed something.
Let's try that and see if it works.
Ok, seems like 0.4.2-BETA-1
works fine. I ran server for 10 hours without players and no new Lootr Minecarts was spawned.
I could test further and play some time on server, but this could take a week.
Does -BETA-1
changes included in 0.4.4
?
clearing out entities before joining the server
Server stall even without players. Its happen after ~10 minutes of server running. Its 3-2 TPS first, then 1 TPS then stall.
Yes, clearing all entities before joining fix stall too. Last time i used /cofh killall *
using /forge entity list to see how many minecarts there
Ok, i will try to use this command next time.
Unless new chunks are generating while there are no players online, there should be no Minecart entities being added to the world...
Have you tried clearing out entities before joining the server, and using /forge entity list
to see how many minecarts there are? It's possible that an excess were in unloaded chunks.
Based on this new information, my next theory is that Entity#isAddedToWorld
behaves differently in 1.12 and somehow it tries to spawn the same minecart numerous times. I'll try looking into this next.
It's strange that, to my knowledge, you are the only person to have this problem so far.
Ok as i said, i removed Lootr to fix stall crash.
Today i installed it back, restarted server and run server for 4 hours. No one joined after restart.
And there is already 600 minecarts, lol
Update:
And its growing! After 10 minutes its +26
carts
Update 2:
Maybe this information unrelevant, but on E2E-E
mineshafts generated with YUNG's Better Mineshafts.
I believe in 1.12.2 you can still use the teleport trick to work out where Minecarts are: try /tp @e[type=lootr:lootr_minecart] ~ ~ ~
and it should print out a list of all the locations of the Minecarts. I'm curious if they're all in or near the same location, or if they're spread out over the map?
If they're all around the same location it could be that a single minecart is getting stuck in the entity ticker and never being cleared... (presuming 1.12's entity ticker uses a similar method of replacement).
Admittedly, as I don't believe there is asynchronous world generation in 1.12, it might be possible to just replace the minecarts at execution rather than deferring them. @embeddedt, thoughts?
Admittedly, as I don't believe there is asynchronous world generation in 1.12, it might be possible to just replace the minecarts at execution rather than deferring them. @embeddedt, thoughts?
That's correct. Let's try that and see if it works.
There is strange thing i encountered.
Server was ran long time, and i came back to see 36000
minecarts. I ran /cofh killall lootr_minecart
, and instead of just remove entities as /cofh killall
usual do, it spawned tons of items. Next call of /forge entities list
show about 1,000,000
of minecraft:item
I wanted to bump this issue as I experienced a similar problem on my server, where I also had Yung's Mineshafts installed. I think, as mentioned above, that with Yung's mods, the server just continually generates chests in their locations and eventually times out the game.
For additional info, this server is in 1.19 and fabric.
Please start a separate issue for this; this issue is for 1.12.2.
I wanted to bump this issue as I experienced a similar problem on my server, where I also had Yung's Mineshafts installed. I think, as mentioned above, that with Yung's mods, the server just continually generates chests in their locations and eventually times out the game.
For additional info, this server is in 1.19 and fabric.