Parallel Masks Causing Deadlock With Brushes
Zeranny opened this issue ยท 1 comments
Server Implementation
Paper
Server Version
1.20
Describe the bug
I hope I haven't gone down the wrong rabbit hole but;
It seems that when using a large enough brush with a mask that checks blocks surrounding the test block (e.g. angle mask) a scenario can arise where two threads will deadlock waiting on the same getChunkLock.lock().
I haven't yet worked out why this is limited to brushes and not also regions.
From VisualVM:
Found one Java-level deadlock:
=============================
"AsyncNotifyKeyedQueue - 0":
waiting for ownable synchronizer 0x00000007d1464f60, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "pool-23-thread-11"
"pool-23-thread-11":
waiting for ownable synchronizer 0x00000007d134bfa0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "AsyncNotifyKeyedQueue - 0"
Java stack information for the threads listed above:
===================================================
"AsyncNotifyKeyedQueue - 0":
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000007d1464f60> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:938)
at java.util.concurrent.locks.ReentrantLock$Sync.lock([email protected]/ReentrantLock.java:153)
at java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:322)
at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.checkAndWaitOnCalledLock(ChunkHolder.java:96)
at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.getBlock(ChunkHolder.java:1128)
at com.fastasyncworldedit.core.queue.IChunkExtent.getBlock(IChunkExtent.java:62)
at com.sk89q.worldedit.extent.Extent.getNearestSurfaceTerrainBlock(Extent.java:445)
at com.sk89q.worldedit.extent.Extent.getNearestSurfaceTerrainBlock(Extent.java:414)
at com.sk89q.worldedit.extent.Extent.getNearestSurfaceTerrainBlock(Extent.java:397)
at com.fastasyncworldedit.core.extent.PassthroughExtent.getNearestSurfaceTerrainBlock(PassthroughExtent.java:67)
at com.fastasyncworldedit.core.extent.PassthroughExtent.getNearestSurfaceTerrainBlock(PassthroughExtent.java:67)
at com.fastasyncworldedit.core.function.mask.AngleMask.getHeight(AngleMask.java:78)
at com.fastasyncworldedit.core.function.mask.AngleMask.testSlope(AngleMask.java:87)
at com.fastasyncworldedit.core.function.mask.AngleMask.test(AngleMask.java:166)
at com.sk89q.worldedit.extent.MaskingExtent.applyBlock(MaskingExtent.java:114)
at com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock.filter(CharFilterBlock.java:176)
- locked <0x00000007d1485058> (a com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock)
at com.fastasyncworldedit.core.extent.filter.block.ChunkFilterBlock.filter(ChunkFilterBlock.java:77)
- locked <0x00000007d1485058> (a com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock)
at com.sk89q.worldedit.extent.MaskingExtent.processSet(MaskingExtent.java:109)
at com.fastasyncworldedit.core.extent.processor.MultiBatchProcessor.processSet(MultiBatchProcessor.java:120)
at com.fastasyncworldedit.core.extent.processor.MultiBatchProcessor.processSet(MultiBatchProcessor.java:106)
at com.fastasyncworldedit.core.extent.processor.BatchProcessorHolder.processSet(BatchProcessorHolder.java:27)
at com.fastasyncworldedit.core.extent.processor.BatchProcessorHolder.processSet(BatchProcessorHolder.java:27)
at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:1075)
- locked <0x00000007d14850c8> (a com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder)
at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:1058)
- locked <0x00000007d14850c8> (a com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder)
at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:34)
at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution([email protected]/ThreadPoolExecutor.java:2037)
at java.util.concurrent.ThreadPoolExecutor.reject([email protected]/ThreadPoolExecutor.java:833)
at java.util.concurrent.ThreadPoolExecutor.execute([email protected]/ThreadPoolExecutor.java:1365)
at java.util.concurrent.AbstractExecutorService.submit([email protected]/AbstractExecutorService.java:145)
at com.fastasyncworldedit.core.queue.implementation.QueueHandler.submit(QueueHandler.java:379)
at com.fastasyncworldedit.core.queue.implementation.SingleThreadQueueExtent.submitUnchecked(SingleThreadQueueExtent.java:251)
at com.fastasyncworldedit.core.queue.implementation.SingleThreadQueueExtent.flush(SingleThreadQueueExtent.java:466)
- locked <0x00000007d1800720> (a com.fastasyncworldedit.core.queue.implementation.SingleThreadQueueExtent)
at com.fastasyncworldedit.core.queue.IQueueExtent.commit(IQueueExtent.java:104)
at com.fastasyncworldedit.core.extent.PassthroughExtent.commit(PassthroughExtent.java:221)
at com.fastasyncworldedit.core.extent.PassthroughExtent.commit(PassthroughExtent.java:221)
at com.sk89q.worldedit.EditSession.flushQueue(EditSession.java:1293)
at com.sk89q.worldedit.LocalSession.remember(LocalSession.java:529)
at com.sk89q.worldedit.LocalSession.remember(LocalSession.java:443)
at com.sk89q.worldedit.command.tool.BrushTool.act(BrushTool.java:485)
at com.sk89q.worldedit.command.tool.BrushTool.actPrimary(BrushTool.java:367)
at com.sk89q.worldedit.extension.platform.PlatformManager.lambda$handlePlayerInput$6(PlatformManager.java:501)
at com.sk89q.worldedit.extension.platform.PlatformManager$$Lambda$8307/0x0000000801e068b8.run(Unknown Source)
at com.sk89q.worldedit.extension.platform.AbstractPlayerActor.lambda$runAction$1(AbstractPlayerActor.java:672)
at com.sk89q.worldedit.extension.platform.AbstractPlayerActor$$Lambda$8261/0x0000000801df8498.run(Unknown Source)
at com.fastasyncworldedit.core.util.task.AsyncNotifyKeyedQueue.lambda$run$0(AsyncNotifyKeyedQueue.java:48)
at com.fastasyncworldedit.core.util.task.AsyncNotifyKeyedQueue$$Lambda$8262/0x0000000801df86b8.call(Unknown Source)
at com.fastasyncworldedit.core.util.task.AsyncNotifyKeyedQueue.lambda$call$1(AsyncNotifyKeyedQueue.java:58)
at com.fastasyncworldedit.core.util.task.AsyncNotifyKeyedQueue$$Lambda$8263/0x0000000801df88d8.call(Unknown Source)
at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
at com.fastasyncworldedit.core.util.task.KeyQueuedExecutorService$KeyRunner.lambda$run$0(KeyQueuedExecutorService.java:150)
at com.fastasyncworldedit.core.util.task.KeyQueuedExecutorService$KeyRunner$$Lambda$8264/0x0000000801df8d00.run(Unknown Source)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec([email protected]/ForkJoinTask.java:1395)
at java.util.concurrent.ForkJoinTask.doExec([email protected]/ForkJoinTask.java:373)
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec([email protected]/ForkJoinPool.java:1182)
at java.util.concurrent.ForkJoinPool.scan([email protected]/ForkJoinPool.java:1655)
at java.util.concurrent.ForkJoinPool.runWorker([email protected]/ForkJoinPool.java:1622)
at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/ForkJoinWorkerThread.java:165)
"pool-23-thread-11":
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000007d134bfa0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:938)
at java.util.concurrent.locks.ReentrantLock$Sync.lock([email protected]/ReentrantLock.java:153)
at java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:322)
at com.fastasyncworldedit.core.queue.implementation.SingleThreadQueueExtent.getOrCreateChunk(SingleThreadQueueExtent.java:291)
at com.fastasyncworldedit.core.queue.implementation.SingleThreadQueueExtent.getOrCreateChunk(SingleThreadQueueExtent.java:46)
at com.fastasyncworldedit.core.queue.IChunkExtent.getBlock(IChunkExtent.java:61)
at com.sk89q.worldedit.extent.InputExtent.getBlock(InputExtent.java:53)
at com.fastasyncworldedit.core.extent.PassthroughExtent.getBlock(PassthroughExtent.java:194)
at com.fastasyncworldedit.core.extent.PassthroughExtent.getBlock(PassthroughExtent.java:194)
at com.sk89q.worldedit.function.mask.BlockMask.test(BlockMask.java:207)
at com.fastasyncworldedit.core.function.mask.CachedMask.test(CachedMask.java:71)
at com.fastasyncworldedit.core.function.mask.AngleMask.test(AngleMask.java:154)
at com.sk89q.worldedit.extent.MaskingExtent.applyBlock(MaskingExtent.java:114)
at com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock.filter(CharFilterBlock.java:176)
- locked <0x00000007d1a32cb8> (a com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock)
at com.fastasyncworldedit.core.extent.filter.block.ChunkFilterBlock.filter(ChunkFilterBlock.java:77)
- locked <0x00000007d1a32cb8> (a com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock)
at com.sk89q.worldedit.extent.MaskingExtent.processSet(MaskingExtent.java:109)
at com.fastasyncworldedit.core.extent.processor.MultiBatchProcessor.processSet(MultiBatchProcessor.java:120)
at com.fastasyncworldedit.core.extent.processor.MultiBatchProcessor.processSet(MultiBatchProcessor.java:106)
at com.fastasyncworldedit.core.extent.processor.BatchProcessorHolder.processSet(BatchProcessorHolder.java:27)
at com.fastasyncworldedit.core.extent.processor.BatchProcessorHolder.processSet(BatchProcessorHolder.java:27)
at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:1075)
- locked <0x00000007d1464fc0> (a com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder)
at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:1058)
- locked <0x00000007d1464fc0> (a com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder)
at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:34)
at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
at java.lang.Thread.run([email protected]/Thread.java:833)
Found 1 deadlock.
To Reproduce
//mask #angle[0d][45d]- Seemingly a mask that checks blocks other than the exact test(BlockVector3) block//brush sphere stone 50- I believe the size increases the odds of the crash.- Use brush
- Server will have deadlocked threads and FAWE will be locked (Paper may also thread dump and close the server - seems to happen if the user that used the brush leaves)
Note that making an equivalent sized region selection and performing the same masked (gmask or replace etc) action will not crash/deadlock.
Expected behaviour
Parallel masks should not enter a thread unsafe deadlock scenario.
Screenshots / Videos
No response
Error log (if applicable)
https://paste.gg/p/anonymous/381e68e542c940208676aa5a76cfcfdc
Fawe Debugpaste
https://athion.net/ISPaster/paste/view/69b8fe8c9eb2437e97aba97df0299cd6
Fawe Version
FastAsyncWorldEdit-Bukkit-2.8.1-SNAPSHOT-574
Checklist
- I have included a Fawe debugpaste.
- I am using the newest build from https://ci.athion.net/job/FastAsyncWorldEdit/ and the issue still persists.
Anything else?
I first encountered this issue on the Bakery Creative server where the Arceon angle mask caused the same crash.
This mask, like the FAWE angle mask, checks blocks surrounding the block being tested.
The parallel masking is a duplicate of #2388 but the actual crash has a different cause so leaving this open