# Apply delay between requests in Elastic Crawler **URL:** \ **Category:** Elastic Search\ **Tags:** crawler\ **Created:** [June 13, 2024, 6:57am UTC](https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375 "2024-06-13T06:57:31Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jahedi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jahedi/32/135254_2.png) [@jahedi](https://discuss.elastic.co/u/jahedi)\ **Post date:** [June 13, 2024, 6:57am UTC](https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375/1 "2024-06-13T06:57:31Z")
Hi, I am using the recently released project , [Elastic Crawler](https://github.com/elastic/crawler), and I can not find any configurations for making some delay between each request while crawling on a domain. Is there any config to set a number to have a specific interval between requests? Thanks ---
**Author:** ![nfeekery](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfeekery/32/117349_2.png) [@nfeekery](https://discuss.elastic.co/u/nfeekery)\ **Post date:** [June 13, 2024, 12:07pm UTC](https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375/2 "2024-06-13T12:07:31Z")
Hi @jahedi! The Open Crawler doesn't currently support delays between crawl requests. We're aware of this gap in functionality so we have this enhancement issue to keep track of it: [Abide by crawl delays found in robots.txt · Issue #46 · elastic/crawler · GitHub](https://github.com/elastic/crawler/issues/46) That issue specifically covers having crawler respect the robots.txt `Crawl-delay` settings, so that would require users have control over the webserver hosting the site. Would this feature resolve your issue? Or do you need a different kind of delay? ---
**Author:** ![nfeekery](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfeekery/32/117349_2.png) [@nfeekery](https://discuss.elastic.co/u/nfeekery)\ **Post date:** [June 13, 2024, 1:57pm UTC](https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375/3 "2024-06-13T13:57:57Z")
Added #crawler ---
**Author:** ![jahedi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jahedi/32/135254_2.png) [@jahedi](https://discuss.elastic.co/u/jahedi)\ **Post date:** [June 13, 2024, 5:13pm UTC](https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375/4 "2024-06-13T17:13:21Z")
Thanks for your response @nfeekery Yes, I had seen this solution with `Crawl-delay` rule in the robots.txt file. But the problem is some websites may not have the robots.txt file or at least may not have the `Crawl-delay` option in the robots.txt file. I was looking for a more general option... ---
**Author:** ![nfeekery](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nfeekery/32/117349_2.png) [@nfeekery](https://discuss.elastic.co/u/nfeekery)\ **Post date:** [June 17, 2024, 7:23am UTC](https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375/5 "2024-06-17T07:23:42Z")
I see how a general delay would be useful. Unfortunately Open Crawler can't do this yet. I've created a new issue to track this: [Add a general delay between crawl requests · Issue #49 · elastic/crawler · GitHub](https://github.com/elastic/crawler/issues/49) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 15, 2024, 7:23am UTC](https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375/6 "2024-07-15T07:23:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.