When adding a new URL to the web crawler the validation fails with this error:
When we fetched the web page at [my domain], it returned data that was not HTML.
curl -I [my domain] from the pod's shell I see that the response is an HTTP 302 with a Location header.
curl [my domain] I get the following text:
<html><body>You are being <a href="[my domain]/[path]">redirected</a>.</body></html>
And if I
curl [my domain]/[path] I get valid HTML.
I'm going to specify the entry point for this domain once its added but I can't get that far in the wizard.
How can I add this site to the crawler?