# \[GitHub Connector\] Handling GitHub API rate limits **URL:** \ **Category:** Elastic Search\ **Tags:** elastic-site-search\ **Created:** [January 30, 2024, 8:24pm UTC](https://discuss.elastic.co/t/github-connector-handling-github-api-rate-limits/352132 "2024-01-30T20:24:39Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![anna-safonov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anna-safonov/32/131045_2.png) [@anna-safonov](https://discuss.elastic.co/u/anna-safonov)\ **Post date:** [January 30, 2024, 8:24pm UTC](https://discuss.elastic.co/t/github-connector-handling-github-api-rate-limits/352132/1 "2024-01-30T20:24:39Z")
Hi there (again), I'm running the GitHub connector on our enterprise instance, and while I am able to do a full sync on smaller repos, we have a large repo that has over 78k+ pull requests, and we hit the rate limit during the full sync. Is there a way to throttle the GraphQL calls the connector makes on full sync? Is this something that we would need to implement ourselves? ---
**Author:** ![Tim\_Grein](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tim_grein/32/121657_2.png) [@Tim\_Grein](https://discuss.elastic.co/u/Tim_Grein)\ **Post date:** [February 5, 2024, 9:14am UTC](https://discuss.elastic.co/t/github-connector-handling-github-api-rate-limits/352132/2 "2024-02-05T09:14:55Z")
Hi Anna, Thank you for your question! The GitHub connector has [rate limiting implemented](https://github.com/elastic/connectors/blob/main/connectors/sources/github.py) and should [throttle the calls to the GraphQL API](https://github.com/elastic/connectors/blob/main/connectors/sources/github.py#L730-L734). May I ask you to provide the versions for Elasticsearch and connectors you're running on? Also do you've any logs similar to [those](https://github.com/elastic/connectors/blob/main/connectors/sources/github.py#L664-L670)? (You must enable DEBUG log level to observe these logs). ---
**Author:** ![anna-safonov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anna-safonov/32/131045_2.png) [@anna-safonov](https://discuss.elastic.co/u/anna-safonov)\ **Post date:** [February 5, 2024, 4:23pm UTC](https://discuss.elastic.co/t/github-connector-handling-github-api-rate-limits/352132/3 "2024-02-05T16:23:45Z")
Hi @Tim_Grein, Thank you for getting back to me! We are running Elasticsearch version 8.11.3, and connector version 8.11.5 built from branch 8.11. Here is what I see in the DEBUG logs for connector service: ```auto [FMWK][11:09:42][INFO] [Connector id: CONNECTOR_ID, index name: search-ghes-drivers, Sync job id: lk8Ceo0BZ1EMbxmageMz] Sync progress -- created: 0 | updated: 9900 | deleted: 0 [FMWK][11:09:42][DEBUG] [Connector id: CONNECTOR_ID, index name: search-ghes-drivers, Sync job id: lk8Ceo0BZ1EMbxmageMz] Sending POST to GHES_SERVER_URL/api/graphql with body: ' [FMWK][11:09:42][DEBUG] Retrying (1 of 3) with interval: 2 and strategy: EXPONENTIAL_BACKOFF [FMWK][11:09:44][DEBUG] [Connector id: CONNECTOR_ID, index name: search-ghes-drivers, Sync job id: lk8Ceo0BZ1EMbxmageMz] Sending POST to GHES_SERVER_URL/api/graphql with body: ' [FMWK][11:09:45][DEBUG] Retrying (2 of 3) with interval: 2 and strategy: EXPONENTIAL_BACKOFF [FMWK][11:09:49][WARNING] [Connector id: CONNECTOR_ID, index name: search-ghes-drivers, Sync job id: lk8Ceo0BZ1EMbxmageMz] Something went wrong while fetching the pull requests. Exception: 406, message='Not Acceptable', url=URL('GHES_SERVER_URL/login?return_to=GHES_SERVER_URL/rate_limit') ``` It looks to me, like it is not waiting for sufficient time to retry (should be 1 hr), but I don't see where to set retry time in the config? or is it hard-coded somewhere in the connector? Piggy-back question 🙂 I see that there is only an option for full syncs scheduled and no incremental sync for GitHub connector, is that right? ---
**Author:** ![Tim\_Grein](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tim_grein/32/121657_2.png) [@Tim\_Grein](https://discuss.elastic.co/u/Tim_Grein)\ **Post date:** [February 6, 2024, 9:27am UTC](https://discuss.elastic.co/t/github-connector-handling-github-api-rate-limits/352132/4 "2024-02-06T09:27:44Z")
The retrying logic is hardcoded in the connectors. This is something we probably want to change in the future and either allow a framework-level or per-connector level configuration. I'll go ahead and create an issue for that, though I cannot guarantee, when and if this will be picked up 🙂. I'm kinda surprised though that this [logic](https://github.com/elastic/connectors/blob/v8.11.0.0/connectors/sources/github.py#L557C15-L566) doesn't kick in correctly. I'll take that with me to the team. The debug logs could also be more useful as we don't see the unit of the interval, that's something I can change directly. > Piggy-back question 🙂 I see that there is only an option for full syncs scheduled and no incremental sync for GitHub connector, is that right? Yes, that's right. You can see [here](https://github.com/elastic/connectors/blob/v8.11.0.0/connectors/sources/github.py#L834-L836) that only advanced rules are implemented. For example [here](https://github.com/elastic/connectors/blob/v8.11.0.0/connectors/sources/sharepoint_online.py#L1136) you can see that incremental syncs are supported for a different connector. Anything else I can help with? And btw feel free to simply create an issue in our repository, if you encounter something is not working, always appreciating feedback and input from the community! ---
**Author:** ![anna-safonov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anna-safonov/32/131045_2.png) [@anna-safonov](https://discuss.elastic.co/u/anna-safonov)\ **Post date:** [February 7, 2024, 8:52pm UTC](https://discuss.elastic.co/t/github-connector-handling-github-api-rate-limits/352132/5 "2024-02-07T20:52:14Z")
Hi @Tim_Grein, I've done some debugging on my own - though I'm not a Python dev, I found my way around relevant code 😃 It seems that the following happens: - the connector makes POST requests while rate limits allow it - it hits the rate limit, executes this [line](https://github.com/elastic/connectors/blob/8.11/connectors/sources/github.py#L639), then goes into [\_put\_to\_sleep](https://github.com/elastic/connectors/blob/8.11/connectors/sources/github.py#L569), and [\_get\_retry\_after](https://github.com/elastic/connectors/blob/8.11/connectors/sources/github.py#L569) from line 570. Inside \_get\_retry\_after, it fails on line 560, because that resolves to another [API call](https://github.com/elastic/connectors/blob/8.11/connectors/sources/github.py#L594C16-L594C25) but the rate is already exceeded at this point, so it fails. I did a rough implementation of throttling the post call, by just running [this](https://github.com/elastic/connectors/blob/8.11/connectors/sources/github.py#L574) for 45 sec before line [624](https://github.com/elastic/connectors/blob/8.11/connectors/sources/github.py#L624). Not the greatest example of dev work, but for now it works, I am able to pull data though it obviously takes some time with over 80k PRs in a repo, but I can look into making it a more elegant solution for our use case, at least. I did try to do the same (i.e. run the same sleep utility function, hardcoded for an hour to reset rate limit) [here](https://github.com/elastic/connectors/blob/8.11/connectors/sources/github.py#L639), but after an hour of waiting, the sync job fails because [this](https://github.com/elastic/connectors/blob/8.11/connectors/sources/github.py#L695) response object resolves to NoneType and thus does not have get attribute. I don't quite understand why that happens, so no idea on how to resolve this, though this would be a preferred solution, I suppose. Another approach would be to monitor rate limit remaining with every post call and to run [this implementation](https://github.com/elastic/connectors/blob/8.11/connectors/sources/github.py#L639) before we run into rate limit. I guess, given different options of going about to fix this, what would be the suggestion from maintainers? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 6, 2024, 8:52pm UTC](https://discuss.elastic.co/t/github-connector-handling-github-api-rate-limits/352132/6 "2024-03-06T20:52:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.