WORLDGUARD-4138 - Reported by andrewkm

This is very weird.
Worldguard tasks got randomly stuck for us.
I've updated/changed nothing with worldguard, and it's the first time I have experienced such an issue in nearly 10 years of using the plugin.

Using any worldguard command such as /rg info spawn:
https://i.imgur.com/BpOWt7p.png

Using /wg running barely shows much at all:
https://i.imgur.com/UWZb1on.png

WorldGuard configuration:
https://pastebin.com/3HgYJZJZ

What could be going on here?
We are not using a MySQL DB so it's not something on that end, we're using simple file system.
As well TPS is perfect 20.0 and there's no lag on the server experiencing this. I'm so confused.

Only way I see to fix this is to server restart, as it won't even let me /wg reload, shoots off same issue with tasks.

Environment:
Paper Build 645 (1.13.2)
WorldEdit 7.0.1 Build 4415
WorldGuard 7.0.1 Build 1969

Comment by andrewkm

Found a way to reproduce, working at figuring out exact issue now :D !

Comment by andrewkm

Alright, here's the deal.
From my reports, /wg running showed:
1st occurence: https://i.imgur.com/UWZb1on.png
2nd occurence: https://i.imgur.com/9fAle2O.png

I noticed both were owner additions to stations.
So I did some investigating on how those regions were setup, and I have reproduced this a good 6+ times, with server restarts, etc. 100% reproduction.

Region spawn:
Has no owners and no members.
Region station1 is a child of region spawn:
Has a member and no owners.

Issue: If adding an owner to a region who's parent has no owners or members, WG locks up 100% of the time.
Fixed the issue by adding an owner to the spawn region, however this is still concerning as it locks up WG and is a potential issue should other parent regions ever become empty.

Comment by andrewkm

A server stop gave me the following:
https://pastebin.com/raw/0dxnPCjy

Upon next start, WG is back to functional, however, this really seems like a very small amount of tasks to simply get stuck. What went on here?
Anyway to cancel pending tasks without a full shutdown in the future?

Comment by wizjany

can you get a thread dump (use jstack) at any time you have a bunch of stuck tasks?
something is blocking them from finishing

someone recently mentioned that a change to using Paper's PlayerProfile cache caused issues for them, but that's not in the 1.13 version of WG, so it seems it's not that.

Comment by andrewkm

Hm, Next time I see this issue I will attempt to get a thread dump.
I have of course at this time restarted (as it is production) to fix the issue.
If I experience it again, I shall let you know.

YOU KNOW BIG BOY it would help if you add me to discord andrewkm#0001 ;) don't worry I'll stroke your belly kindly <3

Comment by andrewkm

Well, 7 days later this issue has not re-appeared.
I will continue to monitor for it to see if we can track down the cause should it happen again.
Feel free to close the ticket for now if you'd like.

Comment by andrewkm

@wizjany it has happened again.
I am just about to restart as players need the WG commands, however as you requested, here is a Jstack:
https://gist.githubusercontent.com/andrewkm/1e36197b2247cb6c62779cd4a726767a/raw/c8eb7c3970b5b0884a4e137acec56b57b6e37c7f/gistfile1.txt
/wg running - shows:
https://i.imgur.com/9fAle2O.png
https://i.imgur.com/bVYBtVX.png
And simply nothing else is going through.
Let me know if you need anymore information.