Cannot crawl a website

Elastic Search
1

Hello,

I'm trying to crawl a website (www.werk.nl) but the crawler gets a 599 error on robots.txt.
If I use my browser, I can open robots.txt and the website without problems.
What can be the issue?

Marten

2

Btw, I'm using AppSearch 7.13.4

3

Hi Marten, a 599 status code can indicate a network connection timeout error. Can you validate that your server is not taking an undue amount of time when responding to App Search's request to robots.txt?

4

Hi Byron,

Please define undue amount.
If I access robots.txt from my browser, I get a response immediately.
In the logging of the crawler I see that the log entry with the status code 599 has exactly the same @timestamp as the log entry with the crawler start, meaning that the the crawler doesn't seem to wait very long for the robots.txt.

Marten

5

I will transfer this issue to the support portal, this one can be closed.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.