# Use NTLM authentication while crawling domains

**URL:** <https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522>\
**Category:** Elastic Search\
**Tags:** elastic-app-search\
**Created:** [March 13, 2023, 6:10am UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522 "2023-03-13T06:10:15Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Disha\_Bodade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/disha_bodade/32/84522_2.png) [@Disha\_Bodade](https://discuss.elastic.co/u/Disha_Bodade)\
**Post date:** [March 13, 2023, 6:10am UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522/1 "2023-03-13T06:10:15Z")

</div>

Hi Team,  
I am trying to crawl website which uses NTLM authentication. but I am not able to crawl it. I can't see any option in UI to add authentication details for website. and also in crawl api we have only basic and raw as a auth type.  
Its not crawling any documents from our website.

Please suggest.

Thanks,  
Disha

---

<div class="post-metadata">

**Author:** ![Chenhui\_Wang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chenhui_wang/32/78869_2.png) [@Chenhui\_Wang](https://discuss.elastic.co/u/Chenhui_Wang)\
**Post date:** [March 13, 2023, 7:04am UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522/2 "2023-03-13T07:04:06Z")

</div>

Hi Disha,

Unfortunately, there's no option to configure auth in UI, and you have to do this via API:  
[https://www.elastic.co/guide/en/app-search/current/web-crawler-reference.html#web-crawler-reference-http-authentication](https://www.elastic.co/guide/en/app-search/current/web-crawler-reference.html#web-crawler-reference-http-authentication)

You can [update the domain](https://www.elastic.co/guide/en/app-search/current/web-crawler-api-reference.html#web-crawler-apis-put-crawler-domain) with auth type raw, and the value will be used directly in the Authorization header (NTLM is one of the supported authentication schemes).

---

<div class="post-metadata">

**Author:** ![Disha\_Bodade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/disha_bodade/32/84522_2.png) [@Disha\_Bodade](https://discuss.elastic.co/u/Disha_Bodade)\
**Post date:** [March 13, 2023, 7:28am UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522/3 "2023-03-13T07:28:56Z")

</div>

Hi Chenhui\_Wang,  
Do I need to configure NTLM server as a http proxy in enterprise\_search.yml like below?

can we use both appSearch authentication and website authorization in same update domain api

```auto
crawler.http.proxy.host: auth.example.com
crawler.http.proxy.port: 443
crawler.http.proxy.protocol: https
crawler.http.proxy.username: username
crawler.http.proxy.password: password

```

---

<div class="post-metadata">

**Author:** ![Disha\_Bodade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/disha_bodade/32/84522_2.png) [@Disha\_Bodade](https://discuss.elastic.co/u/Disha_Bodade)\
**Post date:** [March 14, 2023, 1:27pm UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522/4 "2023-03-14T13:27:21Z")

</div>

Hi Team,  
crawler event logs showing event.type: denied with below message  
Unexpected content type for a crawl task with type=content

what does it means?

---

<div class="post-metadata">

**Author:** ![video](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/video/32/117114_2.png) [@video](https://discuss.elastic.co/u/video)\
**Post date:** [March 20, 2023, 2:17pm UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522/5 "2023-03-20T14:17:35Z")

</div>

Hi @Disha_Bodade,

> Unexpected content type for a crawl task with type=content

When Crawler logs `unexpected content type...` this means it doesn't support or it couldn't recognize the response content-type header. Could you please share the URL if it can be accessed via the public internet or do a basic `curl` command like this:

```auto
curl -i {denied_url}

```

and include the output.

---

<div class="post-metadata">

**Author:** ![video](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/video/32/117114_2.png) [@video](https://discuss.elastic.co/u/video)\
**Post date:** [March 20, 2023, 2:21pm UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522/6 "2023-03-20T14:21:02Z")

</div>

You can use `enterprice_search.yml` however if you configure a proxy server in this way, all your crawlers will be utilizing those proxy properties.

If you want to use proxy configuration per domain, you can use [Crawler API](https://www.elastic.co/guide/en/app-search/current/web-crawler-api-reference.html#web-crawler-apis-put-crawler-domain) to add your configuration.

---

<div class="post-metadata">

**Author:** ![Disha\_Bodade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/disha_bodade/32/84522_2.png) [@Disha\_Bodade](https://discuss.elastic.co/u/Disha_Bodade)\
**Post date:** [April 7, 2023, 6:02pm UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522/7 "2023-04-07T18:02:23Z")

</div>

Our Application team has enabled basic auth for domain, but now also, when I am trying to crawl its showing

```auto
"fetch": {
                "timestamp": "Fri, 07 Apr 2023 16:04:15 +0000",
                "event_id": "64303effe4f766fe7de5b7ff",
                "message": "Unexpected content type for a crawl task with type=content",
                "event_outcome": "failure",
                "duration_msec": 0.00596,
                "http_response": {
                    "status_code": 302,
                    "body_bytes": 0
                },
                "redirect": null
            }

```

Unexpected content type for a crawl task with type=content, I guess some misconfiguration I have done.

---

<div class="post-metadata">

**Author:** ![video](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/video/32/117114_2.png) [@video](https://discuss.elastic.co/u/video)\
**Post date:** [April 11, 2023, 9:53am UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522/8 "2023-04-11T09:53:15Z")

</div>

Have you configured your Crawler with the basic auth credentials?  
If you are using Elastic web crawler - [Managing crawls in Kibana | Elastic Enterprise Search documentation [8.7] | Elastic](https://www.elastic.co/guide/en/enterprise-search/8.7/crawler-managing.html#crawler-managing-authentication).

If you are using App Search Crawler [Web crawler reference | Elastic App Search Documentation [8.7] | Elastic](https://www.elastic.co/guide/en/app-search/current/web-crawler-reference.html#web-crawler-reference-http-authentication)

Could you please verify that your website is accessible via `curl` or similar tools using basic auth?

---

<div class="post-metadata">

**Author:** ![Disha\_Bodade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/disha_bodade/32/84522_2.png) [@Disha\_Bodade](https://discuss.elastic.co/u/Disha_Bodade)\
**Post date:** [April 11, 2023, 10:09am UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522/9 "2023-04-11T10:09:42Z")

</div>

Hi Dimitrii,  
It seems issue with how domain was accepting authentication, as application team setup a proxy and proxy is taking care of authentication, I am able to crawl domain pages properly.

Thanks,  
Disha

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 9, 2023, 10:10am UTC](https://discuss.elastic.co/t/use-ntlm-authentication-while-crawling-domains/327522/10 "2023-05-09T10:10:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.