Checked and made sure that the default group was set correctly and it was.
Moderator is not set as default.

I tried to set the default group to "wanderer" which was working for literally two years, then suddenly is acting up.

I now have hundreds of moderators on my server and all hell has broken loose.

PermissionsEx version 1.23.4 on git-Spigot-078e2f8-7f9fbe5 (MC: 1.10.2) (Implementing API version 1.10.2-R0.1-SNAPSHOT)

No permissions.yml file posted in the past 8 days. I'm guessing this problem is resolved.

Show me your permissions.yml file. Pastebin it.

(Yes, you'll probably have to migrate back to using a file backend to get that.)

Typically, the only way this sort of thing can happen is if:

1. Someone with the power to promote people goes crazy and gives everyone the power, in which case it's exponential and everyone makes everyone else Mods/Admins/Owners too.
2. An inheritance somewhere goes wrong, in which case someone gets 'the power' and gives it to everyone else (as mentioned above).
3. A single user exploits a badly programmed server, especially servers that use OP when they shouldn't be since they use plugins, ends up with 'the power' and gives it to everyone else (as mentioned above).

(So, basically, someone always somehow gets 'the power' and gives it to everyone else.)

In a much more rare case, a server automation plugin like AutoRank or ServerCommands is programmed incorrectly and gives everyone the wrong group at the wrong time. (This is still "someone got 'the power' somehow", but at least it's not their fault this time.)

Alright, I switched to file backend and imported SQL.
I went through all the logs for the past month and made sure nobody was doing anything they shouldn't and came up empty handed.
Permissions.yml shows no default groups that shouldn't be.
It says 'default' is default, but the Wanderer group has the default option set.

It's also worth noting that now players are getting a donor rank along with their Moderator group. (LegendaryMystic). I also see some people who have two of the same group. Seems like something is going horribly wrong.

Uploaded perms file as .txt here because it's too large for any paste site and GitHub doesn't let me upload .yml lol.

permissions.txt

OK, first things first, go through your groups and correct all the weight values. You currently have 6 that are showing negative values and the "best" value that any group can have stops at 1. (This is typically reserved for whatever the server calls the owner of the server's group: Owner, God, King, Zeus, whatever.) The worst value would be something like 1000, and goes to the default: true group. (In your case, whichever single group you decide is getting the default: true setting.) I believe these incorrect weight values may be part of an "everyone owns the server" type of problem. When an admin/owner/plugin uses the promote/demote commands, these incorrect weight settings may be making the users into server owners/etc as you're experiencing. Remember that no 2 groups can have the same weight value unless they're specified as being in 2 different rank-ladders.

I see what you're saying about the "default" and "Wanderer" groups both having default: true. If "Wanderer" is the group you want all newcomers on the server to be members of, use the command /pex group default delete to remove the group named "default" from the server entirely. Anyone who was in the default: true group prior to this deletion should immediately be moved (by PEX) into the "Wanderer" group, since it will be the only default: true group. NOTE: The default: true group isn't actually listed in the users section. PEX knows that any user without a group listed there is in the default: true group, whatever it may be called as there should only be one such group on the server, and this makes it easier for server owners to change what they want the default: true group to be.

Doublecheck the permissions of any and all groups which use wildcards and/or negated permission nodes. All negations need to be moved to the top of the permissions list in order for PEX to find them, apply them correctly, and carry on.

For example, your "Owner" group has the global wildcard - '*' on line 12, and negated permissions coming before and after it. In your current file, this means PEX would see the "Owner" group shouldn't have (line 11) bukkit.command.reload, but will never know that the "Owner" group does not get essentials.remove (line 13) or worldguard.halt-activity (line 14) since PEX will see that the group gets absolutely every permission node on the server when it reads line 12 (that global wildcard), and then stop looking for other permissions rules for that group.

All negated permission nodes should be at the top of the group's permissions:

  Owner:
    permissions:
    - -bukkit.command.reload
    - -essentials.remove
    - -worldguard.halt-activity
    - '*'
    - ''                      <----- This line should be deleted from your file.

Then PEX will see the negations and apply them correctly to the groups and the server.

While you're correcting the negations in each group, check to ensure that their permission nodes actually exist. Line 265 of the "Wanderer" group, for example, has /minecraft:me negated, and that's not actually a permissions node. 'Fake' permission nodes don't usually hamper a server, but if they're really botched or you have thousands of users, they may pose a problem in any number of different ways.

Doublecheck your group names also. You have a group named Joannou1 and MikeyRos (which look to me to be usernames, and possibly people who made everyone mods/admins/owners on the server ... but someone made them into groups), and 2 groups named "coordionator" and "coordinator" (which appear to have been intended to be one group).

You do have 23,373 lines of users on your server, so you may have to use a combination of /pex users cleanup <group> [threshold] (to remove users from the file who have not been on the server for [threshold] number of days; default is 30 days if no threshold is specified) and the Find & Replace function in Notepad or Notepad++ to remove everyone who's erroneously in a Moderator/Admin/Owner type group by finding the Mod/Admin/Owner group names, and replacing them with a lower rank until they log into the game the next time and you can get them fixed again.

After making all of the above corrections, give the server a test again and see if everyone is being turned into mods/admins/owners/etc incorrectly.

From there, we may have to get into a server-wide Debug Mode to find out what else is going on. That will be MASSIVELY spammy in the console, so it would have to be done when virtually no users are on the server.

If at all possible (it's possible), try to keep your groups in somewhat of a hierarchical order as you list them in the permissions.yml file. Newcomers group ("Wanderer", in this case) at the top of the file with a very high weight value, down through all your groups, and finally to the mods (weight of 4), admins (weight of 3), co-owner (weight of 2), and owner groups (with the owner group having a weight of 1). Weight values are suggested; you could just as easily call them 40, 30, 20, and 10 respectively.

You can see this example to more clearly understand the weight values.

Hey, thanks for the in-depth analysis and effort on helping me out here, I really appreciate it!

I've gone through the permissions file and did just about everything you asked, waiting for the results to come in now.
Reorganized perms / order of groups, removed suspect perms from suspect users, redid the weights in the file and removed duplicate groups from users.

For some reason the purge command doesn't work, otherwise I'd purge anyone that is in the wanderer group with no additional perms / options / etc.

So far so good though, deleted alt account from file and rebooted and joined and I'm in the correct group.

I should be able to easily switch backends back and forth without issue, right?
Thanks!

(See attached file for updated perms)
newperms.txt

Good to hear everything seems to be working now. The new perms file looks great too, by the way. You should have no problem switching backends. :)

As for the cleanup command, be sure you're actually typing (for example): /pex users cleanup Wanderer 30 if in-game, and pex users cleanup Wanderer 30 (without the slash) if using the console. (And run it on all of your most popular groups, at least, to clean each of them up.)

If you're getting a message along the lines of access denied, no permission granted, try giving yourself all of the PEX commands directly by typing pex user Joannou1 add permissions.* (or whatever username you use in-game) into the console. That particular command/node (to cleanup) may just be missing from the group you're currently in. This node is good to give yourself directly (as the server owner) so that no matter which group you move yourself to for testing, troubleshooting, and debugging, you'll always have access to all the PEX commands.

Correction: The new perms file has the schema-version: 1 line, which belongs just above the users: section on line 785, in the wrong place. (You have it on line 1 of the file.) THEN the file is great. :)

Alright, sweet, import went well.
Also, I should have been more specific with "it didn't work".. I meant it just cleaned up "0" groups, my bad.

One question though, something I changed made it so that a donor group takes priority over my 'Owner' group despite a larger weight of 500 when the Owner group has a weight of 10. However, said donor group is in its own ladder ('Normal').

Should I move all staff ranks to the Normal ladder instead of the unranked/none currently?
Thanks again!

Since you are using ladders, you could put all of your staff ranks into their own ladder.

How did you discover the donor group was higher priority? I've never heard of that before, especially with the weights set correctly.

I typed in the chat and the donor group prefix overrid the owner group prefix.
I may put them in their own ladder provided it does override the donor ladders.

Are you a member of both groups? If you're in the Owner group, you really wouldn't have any need to be in any other group at the same time.

Yes I was, but this could apply to other staff members who also have donor groups.

That is extremely odd. I've never seen that happen before.

Post your most recent permissions.yml file again and maybe @zml2008 can tell us why something like that would be happening.