Spark thread won't terminate properly on Fabric 1.16.2
Neubulae opened this issue ยท 5 comments
After attaching a profiler to the server the profiler notified me for Spark being potentially deadlocked, and the thread is preventing the server from shutting down after issuing stop
command.
spark-fabric-async-worker
is not a daemon thread, it block JVM shutdown.
Thread dump
Full thread dump OpenJDK 64-Bit Server VM (15.0.1+9 mixed mode, sharing):
Threads class SMR info:
_java_thread_list=0x00007fdeb01ad540, length=24, elements={
0x00007fdf3015b600, 0x00007fdf3015cde0, 0x00007fdf301638a0, 0x00007fdf30165010,
0x00007fdf30166c90, 0x00007fdf30168520, 0x00007fdf30169c80, 0x00007fdf301b0c80,
0x00007fdf301b4b20, 0x00007fdf312fa560, 0x00007fdf31752dc0, 0x00007fdf31533040,
0x00007fdf30084d10, 0x00007fdeb023a500, 0x00007fdeb023ba00, 0x00007fdeb0269c50,
0x00007fdeb026b410, 0x00007fde9c007760, 0x00007fde9c00a590, 0x00007fdeb0625a30,
0x00007fdeb0627a50, 0x00007fde8800c300, 0x00007fdeb0097e40, 0x00007fde8800d5f0
}
"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=7.38ms elapsed=1753.97s tid=0x00007fdf3015b600 nid=0xb waiting on condition [0x00007fdf12360000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList([email protected]/Native Method)
at java.lang.ref.Reference.processPendingReferences([email protected]/Unknown Source)
at java.lang.ref.Reference$ReferenceHandler.run([email protected]/Unknown Source)
"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0.80ms elapsed=1753.97s tid=0x00007fdf3015cde0 nid=0xc in Object.wait() [0x00007fdf1225f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <0x000000009998c748> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove([email protected]/Unknown Source)
- locked <0x000000009998c748> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove([email protected]/Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run([email protected]/Unknown Source)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.17ms elapsed=1753.97s tid=0x00007fdf301638a0 nid=0xd waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #5 daemon prio=9 os_prio=0 cpu=98.94ms elapsed=1753.97s tid=0x00007fdf30165010 nid=0xe runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=62059.86ms elapsed=1753.97s tid=0x00007fdf30166c90 nid=0xf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread0" #8 daemon prio=9 os_prio=0 cpu=7521.67ms elapsed=1753.96s tid=0x00007fdf30168520 nid=0x10 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"Sweeper thread" #9 daemon prio=9 os_prio=0 cpu=866.47ms elapsed=1753.96s tid=0x00007fdf30169c80 nid=0x11 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Notification Thread" #10 daemon prio=9 os_prio=0 cpu=0.10ms elapsed=1753.95s tid=0x00007fdf301b0c80 nid=0x12 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Common-Cleaner" #11 daemon prio=8 os_prio=0 cpu=7.03ms elapsed=1753.95s tid=0x00007fdf301b4b20 nid=0x14 in Object.wait() [0x00007fdf11a33000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <no object reference available>
at java.lang.ref.ReferenceQueue.remove([email protected]/Unknown Source)
- locked <0x00000000999fa540> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run([email protected]/Unknown Source)
at java.lang.Thread.run([email protected]/Unknown Source)
at jdk.internal.misc.InnocuousThread.run([email protected]/Unknown Source)
"Worker-Bootstrap-2" #70 daemon prio=5 os_prio=0 cpu=28672.80ms elapsed=1748.25s tid=0x00007fdf312fa560 nid=0x50 waiting on condition [0x00007fdf1056a000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000000fe0b4cd0> (a java.util.concurrent.ForkJoinPool)
at java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at java.util.concurrent.ForkJoinPool.runWorker([email protected]/Unknown Source)
at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/Unknown Source)
"Timer hack thread" #72 daemon prio=5 os_prio=0 cpu=0.09ms elapsed=1747.12s tid=0x00007fdf31752dc0 nid=0x52 waiting on condition [0x00007fdece232000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep([email protected]/Native Method)
at net.minecraft.class_156$6.run(class_156.java:636)
"Snooper Timer" #77 daemon prio=5 os_prio=0 cpu=0.14ms elapsed=1741.74s tid=0x00007fdf31533040 nid=0x57 in Object.wait() [0x00007fdecd62a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <0x00000000f4d80238> (a java.util.TaskQueue)
at java.lang.Object.wait([email protected]/Unknown Source)
at java.util.TimerThread.mainLoop([email protected]/Unknown Source)
- locked <0x00000000f4d80238> (a java.util.TaskQueue)
at java.util.TimerThread.run([email protected]/Unknown Source)
"DestroyJavaVM" #79 prio=5 os_prio=0 cpu=8060.09ms elapsed=1741.73s tid=0x00007fdf30084d10 nid=0x6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"spark-cpu-monitor" #80 daemon prio=5 os_prio=0 cpu=823.45ms elapsed=1741.18s tid=0x00007fdeb023a500 nid=0x59 waiting on condition [0x00007fdeccbda000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000000f4580e78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"spark-fabric-async-worker" #81 prio=5 os_prio=0 cpu=1.57ms elapsed=1741.18s tid=0x00007fdeb023ba00 nid=0x5a waiting on condition [0x00007fdeccad9000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000000f4581220> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block([email protected]/Unknown Source)
at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown Source)
at java.lang.Thread.run([email protected]/Unknown Source)
"DefaultDispatcher-worker-1" #82 daemon prio=5 os_prio=0 cpu=310.67ms elapsed=1741.08s tid=0x00007fdeb0269c50 nid=0x5b waiting on condition [0x00007fdecd0e4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown Source)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
"DefaultDispatcher-worker-2" #83 daemon prio=5 os_prio=0 cpu=1.19ms elapsed=1741.08s tid=0x00007fdeb026b410 nid=0x5c waiting on condition [0x00007fdeccfe3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown Source)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
"DefaultDispatcher-worker-3" #84 daemon prio=5 os_prio=0 cpu=1.32ms elapsed=1741.07s tid=0x00007fde9c007760 nid=0x5d waiting on condition [0x00007fdeccee2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown Source)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
"DefaultDispatcher-worker-4" #85 daemon prio=5 os_prio=0 cpu=47.52ms elapsed=1741.07s tid=0x00007fde9c00a590 nid=0x5e waiting on condition [0x00007fdecc9d8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown Source)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
"ObjectCleanerThread" #88 daemon prio=1 os_prio=0 cpu=8.48ms elapsed=1739.64s tid=0x00007fdeb0625a30 nid=0x61 in Object.wait() [0x00007fdecd225000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <no object reference available>
at java.lang.ref.ReferenceQueue.remove([email protected]/Unknown Source)
- locked <0x00000000f4800c90> (a java.lang.ref.ReferenceQueue$Lock)
at io.netty.util.internal.ObjectCleaner$1.run(ObjectCleaner.java:54)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run([email protected]/Unknown Source)
"Netty Epoll Server IO #0" #89 daemon prio=5 os_prio=0 cpu=75.41ms elapsed=1739.64s tid=0x00007fdeb0627a50 nid=0x62 runnable [0x00007fde93bee000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run([email protected]/Unknown Source)
"Netty Epoll Server IO #1" #96 daemon prio=5 os_prio=0 cpu=5028.13ms elapsed=1729.49s tid=0x00007fde8800c300 nid=0x6a runnable [0x00007fde934e9000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run([email protected]/Unknown Source)
"ForkJoinPool.commonPool-worker-3" #99 daemon prio=5 os_prio=0 cpu=1.17ms elapsed=1707.11s tid=0x00007fdeb0097e40 nid=0x81 waiting on condition [0x00007fde933e8000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000000ff13e170> (a java.util.concurrent.ForkJoinPool)
at java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
at java.util.concurrent.ForkJoinPool.runWorker([email protected]/Unknown Source)
at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/Unknown Source)
"Netty Epoll Server IO #2" #191 daemon prio=5 os_prio=0 cpu=5860.62ms elapsed=679.77s tid=0x00007fde8800d5f0 nid=0x312 runnable [0x00007fde93aed000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:114)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at java.lang.Thread.run([email protected]/Unknown Source)
"VM Thread" os_prio=0 cpu=257.73ms elapsed=1753.97s tid=0x00007fdf301587f0 nid=0xa runnable
"Shenandoah GC Threads#0" os_prio=0 cpu=43680.35ms elapsed=1753.98s tid=0x00007fdf30094a80 nid=0x7 runnable
"Shenandoah GC Threads#1" os_prio=0 cpu=35459.13ms elapsed=1750.15s tid=0x00007fdf00000f20 nid=0x4e runnable
"Safepoint Cleanup Thread#0" os_prio=0 cpu=108.86ms elapsed=1753.98s tid=0x00007fdf300958e0 nid=0x8 runnable
"VM Periodic Task Thread" os_prio=0 cpu=3393.96ms elapsed=1753.95s tid=0x00007fdf301b2900 nid=0x13 waiting on condition
JNI global refs: 27, weak refs: 0
Heap
Shenandoah Heap
1638M max, 1638M soft max, 310M committed, 305M used
3276 x 512K regions
Status: not cancelled
Reserved region:
- [0x0000000099980000, 0x0000000100000000)
Collection set:
- map (vanilla): 0x0000000000011333
- map (biased): 0x0000000000010000
Metaspace used 86182K, capacity 86898K, committed 88192K, reserved 1126400K
class space used 11461K, capacity 11750K, committed 11904K, reserved 1048576K
spark-fabric-async-worker
is not a daemon thread, it block JVM shutdown.Thread dump
Full thread dump OpenJDK 64-Bit Server VM (15.0.1+9 mixed mode, sharing): Threads class SMR info: _java_thread_list=0x00007fdeb01ad540, length=24, elements={ 0x00007fdf3015b600, 0x00007fdf3015cde0, 0x00007fdf301638a0, 0x00007fdf30165010, 0x00007fdf30166c90, 0x00007fdf30168520, 0x00007fdf30169c80, 0x00007fdf301b0c80, 0x00007fdf301b4b20, 0x00007fdf312fa560, 0x00007fdf31752dc0, 0x00007fdf31533040, 0x00007fdf30084d10, 0x00007fdeb023a500, 0x00007fdeb023ba00, 0x00007fdeb0269c50, 0x00007fdeb026b410, 0x00007fde9c007760, 0x00007fde9c00a590, 0x00007fdeb0625a30, 0x00007fdeb0627a50, 0x00007fde8800c300, 0x00007fdeb0097e40, 0x00007fde8800d5f0 } "Reference Handler" #2 daemon prio=10 os_prio=0 cpu=7.38ms elapsed=1753.97s tid=0x00007fdf3015b600 nid=0xb waiting on condition [0x00007fdf12360000] java.lang.Thread.State: RUNNABLE at java.lang.ref.Reference.waitForReferencePendingList([email protected]/Native Method) at java.lang.ref.Reference.processPendingReferences([email protected]/Unknown Source) at java.lang.ref.Reference$ReferenceHandler.run([email protected]/Unknown Source) "Finalizer" #3 daemon prio=8 os_prio=0 cpu=0.80ms elapsed=1753.97s tid=0x00007fdf3015cde0 nid=0xc in Object.wait() [0x00007fdf1225f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait([email protected]/Native Method) - waiting on <0x000000009998c748> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove([email protected]/Unknown Source) - locked <0x000000009998c748> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove([email protected]/Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run([email protected]/Unknown Source) "Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.17ms elapsed=1753.97s tid=0x00007fdf301638a0 nid=0xd waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Service Thread" #5 daemon prio=9 os_prio=0 cpu=98.94ms elapsed=1753.97s tid=0x00007fdf30165010 nid=0xe runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=62059.86ms elapsed=1753.97s tid=0x00007fdf30166c90 nid=0xf waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task "C1 CompilerThread0" #8 daemon prio=9 os_prio=0 cpu=7521.67ms elapsed=1753.96s tid=0x00007fdf30168520 nid=0x10 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task "Sweeper thread" #9 daemon prio=9 os_prio=0 cpu=866.47ms elapsed=1753.96s tid=0x00007fdf30169c80 nid=0x11 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Notification Thread" #10 daemon prio=9 os_prio=0 cpu=0.10ms elapsed=1753.95s tid=0x00007fdf301b0c80 nid=0x12 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Common-Cleaner" #11 daemon prio=8 os_prio=0 cpu=7.03ms elapsed=1753.95s tid=0x00007fdf301b4b20 nid=0x14 in Object.wait() [0x00007fdf11a33000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait([email protected]/Native Method) - waiting on <no object reference available> at java.lang.ref.ReferenceQueue.remove([email protected]/Unknown Source) - locked <0x00000000999fa540> (a java.lang.ref.ReferenceQueue$Lock) at jdk.internal.ref.CleanerImpl.run([email protected]/Unknown Source) at java.lang.Thread.run([email protected]/Unknown Source) at jdk.internal.misc.InnocuousThread.run([email protected]/Unknown Source) "Worker-Bootstrap-2" #70 daemon prio=5 os_prio=0 cpu=28672.80ms elapsed=1748.25s tid=0x00007fdf312fa560 nid=0x50 waiting on condition [0x00007fdf1056a000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x00000000fe0b4cd0> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source) at java.util.concurrent.ForkJoinPool.runWorker([email protected]/Unknown Source) at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/Unknown Source) "Timer hack thread" #72 daemon prio=5 os_prio=0 cpu=0.09ms elapsed=1747.12s tid=0x00007fdf31752dc0 nid=0x52 waiting on condition [0x00007fdece232000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep([email protected]/Native Method) at net.minecraft.class_156$6.run(class_156.java:636) "Snooper Timer" #77 daemon prio=5 os_prio=0 cpu=0.14ms elapsed=1741.74s tid=0x00007fdf31533040 nid=0x57 in Object.wait() [0x00007fdecd62a000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait([email protected]/Native Method) - waiting on <0x00000000f4d80238> (a java.util.TaskQueue) at java.lang.Object.wait([email protected]/Unknown Source) at java.util.TimerThread.mainLoop([email protected]/Unknown Source) - locked <0x00000000f4d80238> (a java.util.TaskQueue) at java.util.TimerThread.run([email protected]/Unknown Source) "DestroyJavaVM" #79 prio=5 os_prio=0 cpu=8060.09ms elapsed=1741.73s tid=0x00007fdf30084d10 nid=0x6 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "spark-cpu-monitor" #80 daemon prio=5 os_prio=0 cpu=823.45ms elapsed=1741.18s tid=0x00007fdeb023a500 nid=0x59 waiting on condition [0x00007fdeccbda000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x00000000f4580e78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown Source) at java.lang.Thread.run([email protected]/Unknown Source) "spark-fabric-async-worker" #81 prio=5 os_prio=0 cpu=1.57ms elapsed=1741.18s tid=0x00007fdeb023ba00 nid=0x5a waiting on condition [0x00007fdeccad9000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x00000000f4581220> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block([email protected]/Unknown Source) at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown Source) at java.lang.Thread.run([email protected]/Unknown Source) "DefaultDispatcher-worker-1" #82 daemon prio=5 os_prio=0 cpu=310.67ms elapsed=1741.08s tid=0x00007fdeb0269c50 nid=0x5b waiting on condition [0x00007fdecd0e4000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown Source) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) "DefaultDispatcher-worker-2" #83 daemon prio=5 os_prio=0 cpu=1.19ms elapsed=1741.08s tid=0x00007fdeb026b410 nid=0x5c waiting on condition [0x00007fdeccfe3000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown Source) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) "DefaultDispatcher-worker-3" #84 daemon prio=5 os_prio=0 cpu=1.32ms elapsed=1741.07s tid=0x00007fde9c007760 nid=0x5d waiting on condition [0x00007fdeccee2000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown Source) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) "DefaultDispatcher-worker-4" #85 daemon prio=5 os_prio=0 cpu=47.52ms elapsed=1741.07s tid=0x00007fde9c00a590 nid=0x5e waiting on condition [0x00007fdecc9d8000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/Unknown Source) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) "ObjectCleanerThread" #88 daemon prio=1 os_prio=0 cpu=8.48ms elapsed=1739.64s tid=0x00007fdeb0625a30 nid=0x61 in Object.wait() [0x00007fdecd225000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait([email protected]/Native Method) - waiting on <no object reference available> at java.lang.ref.ReferenceQueue.remove([email protected]/Unknown Source) - locked <0x00000000f4800c90> (a java.lang.ref.ReferenceQueue$Lock) at io.netty.util.internal.ObjectCleaner$1.run(ObjectCleaner.java:54) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run([email protected]/Unknown Source) "Netty Epoll Server IO #0" #89 daemon prio=5 os_prio=0 cpu=75.41ms elapsed=1739.64s tid=0x00007fdeb0627a50 nid=0x62 runnable [0x00007fde93bee000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait0(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:114) at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at java.lang.Thread.run([email protected]/Unknown Source) "Netty Epoll Server IO #1" #96 daemon prio=5 os_prio=0 cpu=5028.13ms elapsed=1729.49s tid=0x00007fde8800c300 nid=0x6a runnable [0x00007fde934e9000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait0(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:114) at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at java.lang.Thread.run([email protected]/Unknown Source) "ForkJoinPool.commonPool-worker-3" #99 daemon prio=5 os_prio=0 cpu=1.17ms elapsed=1707.11s tid=0x00007fdeb0097e40 nid=0x81 waiting on condition [0x00007fde933e8000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x00000000ff13e170> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source) at java.util.concurrent.ForkJoinPool.runWorker([email protected]/Unknown Source) at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/Unknown Source) "Netty Epoll Server IO #2" #191 daemon prio=5 os_prio=0 cpu=5860.62ms elapsed=679.77s tid=0x00007fde8800d5f0 nid=0x312 runnable [0x00007fde93aed000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait0(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:114) at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:239) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:256) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at java.lang.Thread.run([email protected]/Unknown Source) "VM Thread" os_prio=0 cpu=257.73ms elapsed=1753.97s tid=0x00007fdf301587f0 nid=0xa runnable "Shenandoah GC Threads#0" os_prio=0 cpu=43680.35ms elapsed=1753.98s tid=0x00007fdf30094a80 nid=0x7 runnable "Shenandoah GC Threads#1" os_prio=0 cpu=35459.13ms elapsed=1750.15s tid=0x00007fdf00000f20 nid=0x4e runnable "Safepoint Cleanup Thread#0" os_prio=0 cpu=108.86ms elapsed=1753.98s tid=0x00007fdf300958e0 nid=0x8 runnable "VM Periodic Task Thread" os_prio=0 cpu=3393.96ms elapsed=1753.95s tid=0x00007fdf301b2900 nid=0x13 waiting on condition JNI global refs: 27, weak refs: 0 Heap Shenandoah Heap 1638M max, 1638M soft max, 310M committed, 305M used 3276 x 512K regions Status: not cancelled Reserved region: - [0x0000000099980000, 0x0000000100000000) Collection set: - map (vanilla): 0x0000000000011333 - map (biased): 0x0000000000010000 Metaspace used 86182K, capacity 86898K, committed 88192K, reserved 1126400K class space used 11461K, capacity 11750K, committed 11904K, reserved 1048576K
huh, then how to daemon it
Should be fixed in the latest version - thanks for reporting (and thanks @james58899 for the PR!)