Cannot crawl a website


I'm trying to crawl a website ( but the crawler gets a 599 error on robots.txt.
If I use my browser, I can open robots.txt and the website without problems.
What can be the issue?


Btw, I'm using AppSearch 7.13.4

Hi Marten, a 599 status code can indicate a network connection timeout error. Can you validate that your server is not taking an undue amount of time when responding to App Search's request to robots.txt?

Hi Byron,

Please define undue amount.
If I access robots.txt from my browser, I get a response immediately.
In the logging of the crawler I see that the log entry with the status code 599 has exactly the same @timestamp as the log entry with the crawler start, meaning that the the crawler doesn't seem to wait very long for the robots.txt.


I will transfer this issue to the support portal, this one can be closed.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.