429 Rate Limits being hit constantly
KsaRedFx opened this issue ยท 52 comments
After running DiscordSRV for... months without issue, at 10pm EST on the 28th our consoles started flooding with messages like this;
[Fri, 29. May 2020 00:04:42 EDT WARN] [JDA] Encountered 429 on route /channels/:channel_id
[Fri, 29. May 2020 00:04:42 EDT WARN] [JDA] Encountered 429 on route /channels/:channel_id
Currently running the latest version from the Spigotmc page.
Unsure why this is happening, perhaps Discord changed their rate limits and for some reason DiscordSRV isn't respecting it?
Seems to happen for both the console, and chat channel bridges.
Developer Edit:
Fix: Update DiscordSRV to 1.19.0
@Macintosh-Fan considering the latest version is 1.19.0, you need to update, preferably to the dev builds at the moment. https://snapshot.discordsrv.com
after an update to 1.19 build, i still have the issue until i decided to stop my vps and update everything on it. I restarted my server (1 bungee and 4 servers) and no more issue. Don't know if this could help but i hope so ;)
The latest snapshot produces... Similar yet slightly different results;
[DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket <SNIP>:guild_id:<SNIP>:webhook_id Retry-After: 147987 ms
I added "Encountered 429 on route" to the list of messages not sent across the Discord Console bridge just to suppress some spam to the console.
And, the latest snapshot has made the error significantly less frequent.
Discord appears to be acting up. As long as you are running the latest version (DiscordSRV v1.18.4 or a snapshot 1.19.0 build), we have no control over this. Nothing to do but wait it out.
Network owners (Those that use Bungeecord) can try splitting up their instances so each backend server is operating under their own token, however given how even individual servers are reporting issues, this likely won't fix it.
Also having this issue, for what it's worth. It's been a really good plugin so far! Appreciate there's not much to be done right now, will attempt KsaRedFx's quick fix of muting the message... as soon as I can figure out how to do that >_<
That can be done by adding "Encountered 429"
in the DiscordConsoleChannelDoNotSendPhrases option and running /discordsrv reload
afterwards.
Wouldnโt recommend it as a long term solution, however. Remove it as soon as discord seems to stop acting up.
Ah you're a star, many thanks! I'll keep tabs and disable once this has blown over, thank you very much =)
Hey, some users reported that using development builds helped with this problem. Please try using a development build from snapshot.discordsrv.com Now in 1.19.0+ https://get.discordsrv.com & react to this message with a ๐ if updating resolved the issue and ๐ if you're still getting the messages after the update
I have to correct myself. The message continues to come. But at longer intervals.
Same problem on our server. Thanks @granny for the tip
JDA has its own rate limiting functionality, perhaps updating the JDA version will fix this issue?
JDA has its own rate limiting functionality, perhaps updating the JDA version will fix this issue?
c590c3e Might've fixed this, please try dev builds again, same url
still getting it. But definitely seems not as frequent as it was with the old build (the latest stable, not the dev build)
With the last dev build, the messages were
May 29 15:49:14 mc1 minecraft[29248]: [15:49:14 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710557529989709867:webhook_id Retry-After: 519087 ms
May 29 15:59:14 mc1 minecraft[29248]: [15:59:14 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710547160331976736:webhook_id Retry-After: 548897 ms
May 29 15:59:14 mc1 minecraft[29248]: [15:59:14 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710557529989709867:webhook_id Retry-After: 519108 ms
With the new build they're
May 29 16:16:39 mc1 minecraft[29685]: [16:16:39 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710557529989709867:webhook_id Retry-After: 74233 ms
May 29 16:26:39 mc1 minecraft[29685]: [16:26:39 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710557529989709867:webhook_id Retry-After: 74354 ms
May 29 16:26:39 mc1 minecraft[29685]: [16:26:39 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:710547160331976736:webhook_id Retry-After: 104104 ms
Note that the Retry-After
values are much lower. So it seems it isn't hitting it as hard.
Oh, and something I just noticed, those messages seem to be 10 minutes apart every time. Is there some sort of scheduled task every 10 minutes that executes a bunch of calls?
it's gotten a lot better now. still a few messages here and there but otherwise it's fixed with the new dev version
I see similar results with the previous dev build from yesterday. Intervals of 10 minutes.
I am waiting for the messages to come through in the latest dev version to confirm.
Can confirm that it is still 10 minute intervals for the message, as phemmer said.
As a side note, I do see that Vanish is no longer respected in the develop versions... But that's for another ticket.
Just to check, those that are still experiencing 429's on latest development builds, do you have webhooks enabled (Experiment_WebhookChatMessageDelivery)?
I am not
Experiment_WebhookChatMessageDelivery: false
Experiment_WebhookChatMessageUsernameFormat: "%displayname%"
Experiment_WebhookChatMessageUsernameFromDiscord: false
KsaRedFx, what's your 429's route? Can you paste the log line here, just wanna check if it's the same one others are getting
I reported the bug, and included my routes above...
But here's the latest in the latest build
[12:57:14 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:591365675357044767:webhook_id Retry-After: 599877 ms
Thanks, so I must be completely on the wrong trail about it having something to do with webhooks. the 'bucket' ending with :webhook_id
made me think that might've been the issue, time to dig deeper
Ah! That makes sense.
Not sure how familiar you are with buckets, but my understanding is buckets are your tokens "Wait Queue" for that uri, they're basically UUIDs for that particular timer, because different channels have different timers. I'm guessing Discord just lazily leaves the placeholder in the string when it does a replace without any data.
Since there's nothing running every 10 minutes, and the PATCH /channels/{channel_id}
would indicate a update to a channel: my guess now is that it might be topic updating (since it runs every 5 minutes, and the ratelimit would be hit on the 2nd attempt).
Please try setting ChannelTopicUpdaterRateInMinutes
to a higher number like 6 minutes to see if the interval between the messages changes to 12 minutes, or goes away completely
Mine appears to be set to -1 at the moment.
I'll let you know if I see anything changed in ~24 minutes.
Setting it to anything below 5 would = 5, as a feature to prevent people from setting it below 5. Disabling it would need to be done by setting the format blank in the messages.yml
Have the problem aswell. Is there any option to just don't log this error till we get a fix? I blows up my console (server) and so I had to deactivate it for now.
While I wouldn't really recommend it:
For the Discord console see granny's response here: #719 (comment)
For the Minecraft console ConsoleSpamFix: https://www.spigotmc.org/resources/console-spam-fix.18410/
Though as a temporary solution, sure
Interestingly enough, despite the fact that experimental websocket is off, I still get
[JDA] Got disconnected from WebSocket (Code: 4000). Attempting to resume session
webhook != websocket, code 4000 is Unknown error
, though ๐ (https://discord.com/developers/docs/topics/opcodes-and-status-codes)
Oh my bad.
Also it's still every 10 minutes.
[13:09:28 INFO]: The DiscordSRV config & lang have been reloaded.
[13:17:15 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:591365675357044767:webhook_id Retry-After: 599897 ms
[13:27:15 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:591365675357044767:webhook_id Retry-After: 599828 ms
[13:37:15 WARN]: [DiscordSRV] [JDA] Encountered 429 on route PATCH/channels/{channel_id} with bucket 9852e1a53c06ffc5a89d65fef85ca4ce:guild_id:591365675357044767:webhook_id Retry-After: 599850 ms
Do we have any confirmation that this is an issue on Discord's side? Their status page says everything is normal, and after ~12 hours I would expect that to have updated if there was an issue. But on the other hand I also can't find any announcements saying they would be adjusting any limits.
And yes, I'm still getting the error after updating to the latest snapshot, though it may be less frequent. it was sporadic before, and haven't been running the snapshot long enough to get a real sense of the prevalence yet.
Edit: Per Vankka's comment, yes, the new error is a 429 on PATCH /channels instead of "429 on route /channels/..."
Discord doesn't announce when they change rate limits, on releases the ratelimit gets hit on /channels/:channel_id where as on dev builds, it's PATCH /channels/:channel_id (and a 'bucket' that's seemingly to do with webhooks). It might not neccaserily be a Discord issue but they most likely changed the ratelimits to be a lot lower than before
You might still run into some messages (especially during start & shutdown (and right after changing the config option)), but while the server is running, you shouldn't run into the messages.
For the message to popup? As long as the remaining time to the next topic updater update was left at the time. When it starts using the new value of 10 minutes, it would only popup when the server starts & stops (and shutdown topics are enabled)
Can be fixed on development builds by setting ChannelTopicUpdaterRateInMinutes
to 10
and running /discord reload
. Development builds (and the upcoming 1.19.0 will) enforce a 10 minute minimum for that now, too
Hey guys, I am too getting this... Have noticed that the Channel topic isn't updating anywhere near as frequent as before... Getting spammed with messages however everything else seems to be functional.
Edit: Ah looks like that was mentioned above too :)
@thecmdradama Get the latest development build from https://snapshot.discordsrv.com
@thecmdradama Get the latest development build from https://snapshot.discordsrv.com
Literally doing that now haha :) Thanks for the hard work
Edit: Looking good, thankyou