The web crawler feature worked great for a Apache web site with no authentication. Once I enabled basic authentication, the web crawler failed. Is there a way to use the web crawler on sites that have authentication enabled or are using SAML SSO?
Following up on this as there have been no replies. Thanks.
Hi @ymoriarty !
Crawling authentication is on the roadmap. A workaround is to let the specific user-agent string for the crawler bypass authentication on the website, if that is possible.
You can set the crawler.http.user_agent in the Enterprise Search configuration. Please take a look at the configuration documentation.
Stay tuned for next releases!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.