# How to customize "crawl\_config" parameters for web crawler

**URL:** <https://discuss.elastic.co/t/how-to-customize-crawl-config-parameters-for-web-crawler/359024>\
**Category:** Elastic Search\
**Tags:** elastic-app-search\
**Created:** [May 8, 2024, 7:53am UTC](https://discuss.elastic.co/t/how-to-customize-crawl-config-parameters-for-web-crawler/359024 "2024-05-08T07:53:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Martin\_BR](https://avatars.discourse-cdn.com/v4/letter/m/87869e/32.png) [@Martin\_BR](https://discuss.elastic.co/u/Martin_BR)\
**Post date:** [May 8, 2024, 7:53am UTC](https://discuss.elastic.co/t/how-to-customize-crawl-config-parameters-for-web-crawler/359024/1 "2024-05-08T07:53:46Z")

</div>

Hi,  
We are currently building a search application based on web crawler.  
We are facing some difficulties to customize the "crawl\_config".

```auto
  "crawl_config": {
    "crawl_rules": [
      {
        "policy": "allow",
        "url_pattern": "***",
        "source": "***"
      }
    ],
    "max_crawl_depth": 10,
    "seed_urls": [
      "***"
    ],
    "sitemap_urls": [],
    "domain_allowlist": [
      "***"
    ],
    "deduplication_settings": [
      {
        "fields": [
          "body_content",
          "domains",
          "headings",
          "links",
          "meta_description",
          "meta_keywords",
          "title"
        ],
        "url_pattern": "***",
        "source": "***"
      }
    ],
    "content_extraction_enabled": false,
    "content_extraction_mime_types": []
  }

```

In the raw json of our crawl we can see a lot of parameters but within the option "crawl with custom settings" we are limited. I imagine that I should change those settings through the API but I was not able to find the related documentation or objects that I need to change. I only found this documentation : [Web crawler API reference | App Search documentation [8.13] | Elastic](https://www.elastic.co/guide/en/app-search/current/web-crawler-api-reference.html#web-crawler-apis-post-crawler-partial-crawl-requests) that seems limited too. For example, it does not address how I can work with "content\_extraction\_mime\_types".

I am new to this part of the elastic stack so any advice/help would be really appreciated.  
Thank you very much.  
Regards,

Martin

---

<div class="post-metadata">

**Author:** ![Sean\_Story](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sean_story/32/69987_2.png) [@Sean\_Story](https://discuss.elastic.co/u/Sean_Story)\
**Post date:** [May 8, 2024, 2:25pm UTC](https://discuss.elastic.co/t/how-to-customize-crawl-config-parameters-for-web-crawler/359024/2 "2024-05-08T14:25:59Z")

</div>

Hi @Martin_BR,

Unfortunately, the Elastic Web Crawler does not have public APIs today. We're aware that this is a significant limitation, and we're working to improve this, so stay tuned in the coming months.

Today, the mime\_types for content extraction can't be controlled crawl-to-crawl or even index-to-index/engine-to-engine. Instead, they're set deployment wide via a YAML configuration in enterprise-search.yml. See: [Crawler Binary Content Extraction](https://www.elastic.co/guide/en/enterprise-search/current/crawler-managing.html#crawler-managing-binary-content).

---

<div class="post-metadata">

**Author:** ![Martin\_BR](https://avatars.discourse-cdn.com/v4/letter/m/87869e/32.png) [@Martin\_BR](https://discuss.elastic.co/u/Martin_BR)\
**Post date:** [May 8, 2024, 2:39pm UTC](https://discuss.elastic.co/t/how-to-customize-crawl-config-parameters-for-web-crawler/359024/3 "2024-05-08T14:39:10Z")

</div>

Thank you very much for your answer Sean!  
I am looking forward for the future versions 🙂  
This can be closed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 5, 2024, 2:40pm UTC](https://discuss.elastic.co/t/how-to-customize-crawl-config-parameters-for-web-crawler/359024/4 "2024-06-05T14:40:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.