would you be willing to share the method you use to get the data from wcl?

If my server-side solution was more clean and documented I'd invite you in right away :D

Before sharing the code I would certainly have to make sure all my credentials are safe and at least the core workflows are documented, which is not an easy task. Especially the git publishing method is, well, let's call it not that sophisticated.

I'm currently at almost 1.5 million players and with my latest rework for update queues I expect it to double in the next 2-3 weeks.

Mind if I ask what issue made you work on your own grabber?

I'm using the official warcraft logs api (https://www.warcraftlogs.com/v2-api-docs/warcraft/) roughly like this:

• Scan for new reports (GraphQL Report) every few minutes using the filters startTime, endTime and zoneId
  • Only scan for reports that are at least 12 hours old to avoid still active live-logs
  • Only include reports for relevant raids to reduce the required api calls
  • Add all players within the matching reports into a update queue (including the information which raid was logged)
• Update the player logs (GraphQL Character) using the zoneRankings
  • If the player is not known at all, do a full update (all raid for both 10 and 25 players)
  • If the player is already known, do a partial update using the information from the queue (if a naxx25 report was scanned e.g. only update that)

Obviously thats a simplification, but should give a rough idea of my approach. The most important thing is reducing the number of api calls as far as possible and being a patreon for more api calls xD