DiscordSRV

DiscordSRV

86.8k Downloads

429 Rate Limits being hit constantly

KsaRedFx opened this issue ยท 52 comments

commented

After running DiscordSRV for... months without issue, at 10pm EST on the 28th our consoles started flooding with messages like this;

[Fri, 29. May 2020 00:04:42 EDT WARN] [JDA] Encountered 429 on route /channels/:channel_id
[Fri, 29. May 2020 00:04:42 EDT WARN] [JDA] Encountered 429 on route /channels/:channel_id

Currently running the latest version from the Spigotmc page.
Unsure why this is happening, perhaps Discord changed their rate limits and for some reason DiscordSRV isn't respecting it?

Seems to happen for both the console, and chat channel bridges.


Developer Edit:

Fix: Update DiscordSRV to 1.19.0

commented

I still have this issue.
ConfigVersion: 1.18.4

commented

@Macintosh-Fan considering the latest version is 1.19.0, you need to update, preferably to the dev builds at the moment. https://snapshot.discordsrv.com

commented

after an update to 1.19 build, i still have the issue until i decided to stop my vps and update everything on it. I restarted my server (1 bungee and 4 servers) and no more issue. Don't know if this could help but i hope so ;)

commented

The latest snapshot produces... Similar yet slightly different results;

[DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket <SNIP>:guild_id:<SNIP>:webhook_id Retry-After: 147987 ms
commented

I am having this issue as well. Will await a response from the dev(s).

commented

i have same problem lol

commented

I added "Encountered 429 on route" to the list of messages not sent across the Discord Console bridge just to suppress some spam to the console.

And, the latest snapshot has made the error significantly less frequent.

commented

Discord appears to be acting up. As long as you are running the latest version (DiscordSRV v1.18.4 or a snapshot 1.19.0 build), we have no control over this. Nothing to do but wait it out.

Network owners (Those that use Bungeecord) can try splitting up their instances so each backend server is operating under their own token, however given how even individual servers are reporting issues, this likely won't fix it.

commented

Also having this issue, for what it's worth. It's been a really good plugin so far! Appreciate there's not much to be done right now, will attempt KsaRedFx's quick fix of muting the message... as soon as I can figure out how to do that >_<

commented

That can be done by adding "Encountered 429" in the DiscordConsoleChannelDoNotSendPhrases option and running /discordsrv reload afterwards.

Wouldnโ€™t recommend it as a long term solution, however. Remove it as soon as discord seems to stop acting up.

commented

Ah you're a star, many thanks! I'll keep tabs and disable once this has blown over, thank you very much =)

commented

Hey, some users reported that using development builds helped with this problem. Please try using a development build from snapshot.discordsrv.com Now in 1.19.0+ https://get.discordsrv.com & react to this message with a ๐Ÿ‘ if updating resolved the issue and ๐Ÿ‘Ž if you're still getting the messages after the update

commented

Having this issue as well.

commented

I have to correct myself. The message continues to come. But at longer intervals.

commented

Same problem on our server. Thanks @granny for the tip

commented

JDA has its own rate limiting functionality, perhaps updating the JDA version will fix this issue?

commented

JDA has its own rate limiting functionality, perhaps updating the JDA version will fix this issue?

commented

c590c3e Might've fixed this, please try dev builds again, same url

commented

still getting it. But definitely seems not as frequent as it was with the old build (the latest stable, not the dev build)

commented

how many times, is it still frequent?

commented

With the last dev build, the messages were

May 29 15:49:14 mc1 minecraft[29248]: [15:49:14 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710557529989709867:webhook_id Retry-After: 519087 ms
May 29 15:59:14 mc1 minecraft[29248]: [15:59:14 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710547160331976736:webhook_id Retry-After: 548897 ms
May 29 15:59:14 mc1 minecraft[29248]: [15:59:14 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710557529989709867:webhook_id Retry-After: 519108 ms

With the new build they're

May 29 16:16:39 mc1 minecraft[29685]: [16:16:39 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710557529989709867:webhook_id Retry-After: 74233 ms
May 29 16:26:39 mc1 minecraft[29685]: [16:26:39 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710557529989709867:webhook_id Retry-After: 74354 ms
May 29 16:26:39 mc1 minecraft[29685]: [16:26:39 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710547160331976736:webhook_id Retry-After: 104104 ms

Note that the Retry-After values are much lower. So it seems it isn't hitting it as hard.

commented

Oh, and something I just noticed, those messages seem to be 10 minutes apart every time. Is there some sort of scheduled task every 10 minutes that executes a bunch of calls?

commented

it's gotten a lot better now. still a few messages here and there but otherwise it's fixed with the new dev version

commented

I see similar results with the previous dev build from yesterday. Intervals of 10 minutes.
I am waiting for the messages to come through in the latest dev version to confirm.

commented

Can confirm that it is still 10 minute intervals for the message, as phemmer said.

As a side note, I do see that Vanish is no longer respected in the develop versions... But that's for another ticket.

commented

Just to check, those that are still experiencing 429's on latest development builds, do you have webhooks enabled (Experiment_WebhookChatMessageDelivery)?

commented

I am not

Experiment_WebhookChatMessageDelivery: false
Experiment_WebhookChatMessageUsernameFormat: "%displayname%"
Experiment_WebhookChatMessageUsernameFromDiscord: false
commented

KsaRedFx, what's your 429's route? Can you paste the log line here, just wanna check if it's the same one others are getting

commented

I reported the bug, and included my routes above...
But here's the latest in the latest build

[12:57:14 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:591365675357044767:webhook_id Retry-After: 599877 ms
commented

Thanks, so I must be completely on the wrong trail about it having something to do with webhooks. the 'bucket' ending with :webhook_id made me think that might've been the issue, time to dig deeper

commented

Ah! That makes sense.

Not sure how familiar you are with buckets, but my understanding is buckets are your tokens "Wait Queue" for that uri, they're basically UUIDs for that particular timer, because different channels have different timers. I'm guessing Discord just lazily leaves the placeholder in the string when it does a replace without any data.

commented

Since there's nothing running every 10 minutes, and the PATCH /channels/{channel_id} would indicate a update to a channel: my guess now is that it might be topic updating (since it runs every 5 minutes, and the ratelimit would be hit on the 2nd attempt).
Please try setting ChannelTopicUpdaterRateInMinutes to a higher number like 6 minutes to see if the interval between the messages changes to 12 minutes, or goes away completely

commented

Mine appears to be set to -1 at the moment.
I'll let you know if I see anything changed in ~24 minutes.

commented

Setting it to anything below 5 would = 5, as a feature to prevent people from setting it below 5. Disabling it would need to be done by setting the format blank in the messages.yml

commented

Have the problem aswell. Is there any option to just don't log this error till we get a fix? I blows up my console (server) and so I had to deactivate it for now.

commented

While I wouldn't really recommend it:
For the Discord console see granny's response here: #719 (comment)
For the Minecraft console ConsoleSpamFix: https://www.spigotmc.org/resources/console-spam-fix.18410/
Though as a temporary solution, sure

commented

Interestingly enough, despite the fact that experimental websocket is off, I still get
[JDA] Got disconnected from WebSocket (Code: 4000). Attempting to resume session

commented

webhook != websocket, code 4000 is Unknown error, though ๐Ÿ™„ (https://discord.com/developers/docs/topics/opcodes-and-status-codes)

commented

Oh my bad.

Also it's still every 10 minutes.

[13:09:28 INFO]: The DiscordSRV config & lang have been reloaded.
[13:17:15 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:591365675357044767:webhook_id Retry-After: 599897 ms
[13:27:15 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:591365675357044767:webhook_id Retry-After: 599828 ms
[13:37:15 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:591365675357044767:webhook_id Retry-After: 599850 ms
commented

Do we have any confirmation that this is an issue on Discord's side? Their status page says everything is normal, and after ~12 hours I would expect that to have updated if there was an issue. But on the other hand I also can't find any announcements saying they would be adjusting any limits.

And yes, I'm still getting the error after updating to the latest snapshot, though it may be less frequent. it was sporadic before, and haven't been running the snapshot long enough to get a real sense of the prevalence yet.

Edit: Per Vankka's comment, yes, the new error is a 429 on PATCH /channels instead of "429 on route /channels/..."

commented

Discord doesn't announce when they change rate limits, on releases the ratelimit gets hit on /channels/:channel_id where as on dev builds, it's PATCH /channels/:channel_id (and a 'bucket' that's seemingly to do with webhooks). It might not neccaserily be a Discord issue but they most likely changed the ratelimits to be a lot lower than before

commented

But get still this messages. Do I need to use the latest dev build?

commented

You might still run into some messages (especially during start & shutdown (and right after changing the config option)), but while the server is running, you shouldn't run into the messages.

commented

how long does "right after changing" usually take?

commented

For the message to popup? As long as the remaining time to the next topic updater update was left at the time. When it starts using the new value of 10 minutes, it would only popup when the server starts & stops (and shutdown topics are enabled)

commented

Okay, never mind...
The messages just stopped to spam in the console.

commented

Update: looks like it is to do with the Topic Updater from my tests

commented

So... that means what? can you easily remove/disable your tests?

commented

Can be fixed on development builds by setting ChannelTopicUpdaterRateInMinutes to 10 and running /discord reload. Development builds (and the upcoming 1.19.0 will) enforce a 10 minute minimum for that now, too

commented

Hey guys, I am too getting this... Have noticed that the Channel topic isn't updating anywhere near as frequent as before... Getting spammed with messages however everything else seems to be functional.
Edit: Ah looks like that was mentioned above too :)

commented

@thecmdradama Get the latest development build from https://snapshot.discordsrv.com

commented

@thecmdradama Get the latest development build from https://snapshot.discordsrv.com

Literally doing that now haha :) Thanks for the hard work
Edit: Looking good, thankyou