# Throttling of Elastic Web Crawler

**URL:** <https://discuss.elastic.co/t/throttling-of-elastic-web-crawler/342955>\
**Category:** Elastic Search\
**Tags:** elastic-app-search\
**Created:** [September 13, 2023, 1:18pm UTC](https://discuss.elastic.co/t/throttling-of-elastic-web-crawler/342955 "2023-09-13T13:18:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sebastianboelling](https://avatars.discourse-cdn.com/v4/letter/s/258eb7/32.png) [@sebastianboelling](https://discuss.elastic.co/u/sebastianboelling)\
**Post date:** [September 13, 2023, 1:18pm UTC](https://discuss.elastic.co/t/throttling-of-elastic-web-crawler/342955/1 "2023-09-13T13:18:00Z")

</div>

Hi,

we are using the Elastic Web Crawler. Is there a way or an idea to limit or throttle the requests the Crawler makes againts a Web datasource (a domain)? For example max 1 request per second ro wait x miliseconds between requests ...

Best regards

Sebastian

---

<div class="post-metadata">

**Author:** ![Sean\_Story](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sean_story/32/69987_2.png) [@Sean\_Story](https://discuss.elastic.co/u/Sean_Story)\
**Post date:** [September 13, 2023, 1:31pm UTC](https://discuss.elastic.co/t/throttling-of-elastic-web-crawler/342955/2 "2023-09-13T13:31:08Z")

</div>

Unfortunately, the crawler doesn't currently support the robots.txt directive for `crawl-delay`, which would be a common way to do what you're looking for. If you have a support relationship with Elastic, you can request that feature as an Enhancement Request.

Other than that, you can try to just reduce the parallelism with which the crawler operates, to try to reduce load on your site. Configs like:

- `connector.crawler.workers.pool_size.limit`
- `connector.crawler.crawl.threads.limit`

can help with this.

Finally, ensuring that your site responds with 429 errors with the `Retry-After` header can ensure that the crawler does not over-saturate your request pool.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2023, 1:31pm UTC](https://discuss.elastic.co/t/throttling-of-elastic-web-crawler/342955/3 "2023-10-11T13:31:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.