Web Crawler Failed HTTP request: Unable to request "< domain >" because it resolved to only private/invalid addresses

[jerrac (David Reagan)]

When I try to run the web crawler against a site we host, it fails with this error:

Failed HTTP request: Unable to request "< domain >" because it resolved to only private/invalid addresses

The site in question would resolve to a 10.n.n.n ip address. Is the crawler configured to reject that? Is there a way to override that behavior?

[jerrac (David Reagan)]

Not sure it's related, but if I target my personal site, not hosted internally, it fails as well.

In the logs I see:

Allow none because robots.txt responded with status 599

and

Failed HTTP request: Remote host terminated the handshake

That also happens if I target Elastic Blog: Stories, Tutorials, Releases | Elastic Blog.

I double checked my personal site's robot.txt file. It's the default Drupal 8 robots.txt file. So there shouldn't be anything in it that would completely block the crawler.

Anyway, I'm glad this still beta.

[orhantoy]

@jerrac

Is the crawler configured to reject that? Is there a way to override that behavior?

Yes, that's the current, default behavior and it will become configurable in the next minor release.

As for the other issue you're experiencing, it sounds like you can't crawl any site at all, is that correct?

[jerrac (David Reagan)]

@orhantoy

Yes, that's the current, default behavior and it will become configurable in the next minor release.

Nice.

@orhantoy

As for the other issue you're experiencing, it sounds like you can't crawl any site at all, is that correct?

Yep, can't crawl my personal site, or Elastic Blog: Stories, Tutorials, Releases | Elastic Blog. Haven't tried any other sites yet.