# Web crawler Error: Allow none because robots.txt responded with status 599, read\_timeout

**URL:** <https://discuss.elastic.co/t/web-crawler-error-allow-none-because-robots-txt-responded-with-status-599-read-timeout/359611>\
**Category:** Elastic Cloud Enterprise (ECE)\
**Created:** [May 16, 2024, 9:34am UTC](https://discuss.elastic.co/t/web-crawler-error-allow-none-because-robots-txt-responded-with-status-599-read-timeout/359611 "2024-05-16T09:34:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sivagurlinka](https://avatars.discourse-cdn.com/v4/letter/s/a6a055/32.png) [@sivagurlinka](https://discuss.elastic.co/u/sivagurlinka)\
**Post date:** [May 16, 2024, 9:34am UTC](https://discuss.elastic.co/t/web-crawler-error-allow-none-because-robots-txt-responded-with-status-599-read-timeout/359611/1 "2024-05-16T09:34:57Z")

</div>

Enterprise search web crawler is giving the below error in logs  
`Allow none because robots.txt responded with status 599`  
`Error: read_timeout`

what could be the potential issue and Please suggest resolution.

I see the below exceptions while adding domain. I can able to access, domain URL and robots.txt file through browser.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/7/c73960b65c81ecd5c7e2f81835ae78881764264a.png)

This issue is happening for one specific public domain only, I am able to crawl other public websites

Below is EES config

```auto

allow_es_settings_modification: true
elasticsearch.host: https://xxxxxxxxxxxxx:9200
elasticsearch.ssl.enabled: true
elasticsearch.ssl.verify: false
kibana.host: http://xxxxxxxx:5601
ent_search.listen_host: 0.0.0.0
ent_search.listen_port: 3002

connector.crawler.http.proxy.host: xxxxxxxxxxxx
connector.crawler.http.proxy.port: 80
connector.crawler.http.proxy.protocol: http
connector.crawler.security.dns.allow_private_networks_access: true
connector.crawler.security.dns.allow_loopback_access: true
connector.crawler.content_extraction.enabled: true
connector.crawler.content_extraction.mime_types: ["application/pdf", "application/msword", "text/plain", "application/xml", "text/html", "text/css"]

crawler.http.proxy.host: xxxxxxxxxxxx
crawler.http.proxy.port: 80
crawler.http.proxy.protocol: http
crawler.security.dns.allow_loopback_access: true
crawler.security.dns.allow_private_networks_access: true
crawler.content_extraction.enabled: true
crawler.content_extraction.mime_types: ["application/pdf", "application/msword", "text/plain", "application/xml", "text/html", "text/css"]

ent_search.ssl.enabled: false
crawler.security.ssl.verification_mode: none
connector.crawler.security.ssl.verification_mode: none
crawler.http.request_timeout: 90
crawler.http.read_timeout: 30

```