Enterprise Search Web Crawler fails on authn enabled websites

The web crawler feature worked great for a Apache web site with no authentication. Once I enabled basic authentication, the web crawler failed. Is there a way to use the web crawler on sites that have authentication enabled or are using SAML SSO?

Following up on this as there have been no replies. Thanks.

Hi @ymoriarty !

Crawling authentication is on the roadmap. A workaround is to let the specific user-agent string for the crawler bypass authentication on the website, if that is possible.

You can set the crawler.http.user_agent in the Enterprise Search configuration. Please take a look at the configuration documentation.

Stay tuned for next releases!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.