# \#crawler **URL:** https://discuss.elastic.co/tag/crawler/148.md [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) --- ## [Elastic Open Crawler 0.3.0 - /var/lib/docker/overlay filling up](https://discuss.elastic.co/t/elastic-open-crawler-0-3-0-var-lib-docker-overlay-filling-up/379635)
**Author:** [@alongaks](https://discuss.elastic.co/u/alongaks)\ **Replies:** 7\ **Last updated:** [July 1, 2025, 3:45pm UTC](https://discuss.elastic.co/t/elastic-open-crawler-0-3-0-var-lib-docker-overlay-filling-up/379635 "2025-07-01T15:45:14Z")
Hello, I have been working with the Open Crawler for a bit, trying to tune it to replace the Enterprise Search crawler at some point. While running crawls I began seeing the following errors in logs and the resultant d… --- ## [Crawl sitemap only](https://discuss.elastic.co/t/crawl-sitemap-only/375169)
**Author:** [@pngworkforce](https://discuss.elastic.co/u/pngworkforce)\ **Replies:** 7\ **Last updated:** [March 20, 2025, 7:53am UTC](https://discuss.elastic.co/t/crawl-sitemap-only/375169 "2025-03-20T07:53:55Z")
Hello! I have a question similar to the one listed here -\> Web crawler is crawling URLs that are not on the sitemap I have added a sitemap index as the sitemap to my crawler in Elastic Cloud UI. How do I instruct the … --- ## [What happens to existing documents during a crawl?](https://discuss.elastic.co/t/what-happens-to-existing-documents-during-a-crawl/374366)
**Author:** [@jkyeusun](https://discuss.elastic.co/u/jkyeusun)\ **Replies:** 1\ **Last updated:** [February 11, 2025, 1:53pm UTC](https://discuss.elastic.co/t/what-happens-to-existing-documents-during-a-crawl/374366 "2025-02-11T13:53:56Z")
For the current version of Enterprise Search (8.17), I can't find any documentation on how already indexed documents are managed in a web crawler index, so here are my questions regarding that. So far I've been looking … --- ## [Force a full recrawl](https://discuss.elastic.co/t/force-a-full-recrawl/370484)
**Author:** [@pngworkforce](https://discuss.elastic.co/u/pngworkforce)\ **Replies:** 4\ **Last updated:** [November 21, 2024, 12:41am UTC](https://discuss.elastic.co/t/force-a-full-recrawl/370484 "2024-11-21T00:41:00Z")
Hello! Is there a way to force a full recrawl of all documents in the Elastic Cloud UI? We have added some mapped fields but they are not applied to docs indexed before the mapped fields were added. I read that a rein… --- ## [Elastic Cloud - Export Index with Crawler](https://discuss.elastic.co/t/elastic-cloud-export-index-with-crawler/370189)
**Author:** [@pngworkforce](https://discuss.elastic.co/u/pngworkforce)\ **Replies:** 2\ **Last updated:** [November 13, 2024, 2:29pm UTC](https://discuss.elastic.co/t/elastic-cloud-export-index-with-crawler/370189 "2024-11-13T14:29:45Z")
Hello!, Is there a way to export an index's schema and crawler configuration on Elastic Cloud? We are looking to create some workflows in Git and would like to be able to store a JSON schema to deploy an index with cra… --- ## [Elastic crawler metadata content extraction](https://discuss.elastic.co/t/elastic-crawler-metadata-content-extraction/369160)
**Author:** [@pngworkforce](https://discuss.elastic.co/u/pngworkforce)\ **Replies:** 2\ **Last updated:** [October 21, 2024, 3:03pm UTC](https://discuss.elastic.co/t/elastic-crawler-metadata-content-extraction/369160 "2024-10-21T15:03:21Z")
Hello! Is there a way to extract html metadata fields with the elastic crawler without setting the class=“elastic” on them? We have inherited a large flat file html site we would like to index and it would take signifi… --- ## [Ignoring robots noindex / nofollow in Elastic crawler](https://discuss.elastic.co/t/ignoring-robots-noindex-nofollow-in-elastic-crawler/369104)
**Author:** [@pngworkforce](https://discuss.elastic.co/u/pngworkforce)\ **Replies:** 3\ **Last updated:** [October 21, 2024, 2:57pm UTC](https://discuss.elastic.co/t/ignoring-robots-noindex-nofollow-in-elastic-crawler/369104 "2024-10-21T14:57:48Z")
Hello! Is there a way to ignore the meta robots nofollow / noindex in the Elastic crawler settings? If not, is there some other way to filter these meta fields out? We have a development site with these robots meta ta… --- ## [Web crawler fields indexed without position data; cannot run PhraseQuery](https://discuss.elastic.co/t/web-crawler-fields-indexed-without-position-data-cannot-run-phrasequery/366758)
**Author:** [@sarahg](https://discuss.elastic.co/u/sarahg)\ **Replies:** 9\ **Last updated:** [September 26, 2024, 8:54pm UTC](https://discuss.elastic.co/t/web-crawler-fields-indexed-without-position-data-cannot-run-phrasequery/366758 "2024-09-26T20:54:54Z")
Hello, I'm getting started setting up Elasticsearch for a technical documentation website. We are using this stack: Elastic Cloud Indexing via the Web Crawler Search UI for the frontend I'm having a hard time getting… --- ## [API Interface for Elastic Search Index](https://discuss.elastic.co/t/api-interface-for-elastic-search-index/366817)
**Author:** [@raylowe](https://discuss.elastic.co/u/raylowe)\ **Replies:** 3\ **Last updated:** [September 19, 2024, 6:42pm UTC](https://discuss.elastic.co/t/api-interface-for-elastic-search-index/366817 "2024-09-19T18:42:07Z")
Hi, I am having some trouble with the API interface for the Elasticsearch web crawler. Background: I have a created an Elastic search index with a web crawler. I have also created an engine using the "Elasticsearch i… --- ## [Does FSCrawler support chunking?](https://discuss.elastic.co/t/does-fscrawler-support-chunking/365837)
**Author:** [@Santiago\_Rubio](https://discuss.elastic.co/u/Santiago_Rubio)\ **Replies:** 7\ **Last updated:** [September 6, 2024, 5:10pm UTC](https://discuss.elastic.co/t/does-fscrawler-support-chunking/365837 "2024-09-06T17:10:46Z")
Hey all! Hope you are doing great. I've recently started working on a solution using Elasticsearch and we have the need to parse and upload different kinds of documents, such as emails, ppts, pdfs, etc. The client reque… --- ## [Can't get extraction rulesets working](https://discuss.elastic.co/t/cant-get-extraction-rulesets-working/363892)
**Author:** [@CobusT](https://discuss.elastic.co/u/CobusT)\ **Replies:** 5\ **Last updated:** [July 30, 2024, 2:56pm UTC](https://discuss.elastic.co/t/cant-get-extraction-rulesets-working/363892 "2024-07-30T14:56:28Z")
I am trying to configure the crawler to extract content from a specific selector in the html documents (.content-container) into an ES field called my\_content. When I run the crawler (v0.2.0) against ES (v.8.14.3) I see b… --- ## [Ghost Web Crawlers](https://discuss.elastic.co/t/ghost-web-crawlers/363373)
**Author:** [@mszal\_ib](https://discuss.elastic.co/u/mszal_ib)\ **Replies:** 3\ **Last updated:** [July 19, 2024, 8:20pm UTC](https://discuss.elastic.co/t/ghost-web-crawlers/363373 "2024-07-19T20:20:59Z")
I am using Elastic, Kibana, and Enterprise Search 8.14 on Elastic Cloud. I was test crawling some domains with the "Elastic Web Crawler" with multiple custom crawl schedules (top right drop down Kibana). Later, I deleted… --- ## [Enterprise Search encountered an internal server error](https://discuss.elastic.co/t/enterprise-search-encountered-an-internal-server-error/362658)
**Author:** [@elitzur\_e](https://discuss.elastic.co/u/elitzur_e)\ **Replies:** 1\ **Last updated:** [July 9, 2024, 10:28am UTC](https://discuss.elastic.co/t/enterprise-search-encountered-an-internal-server-error/362658 "2024-07-09T10:28:34Z")
we have been seeing: Enterprise Search encountered an internal server error. Please contact your system administrator if the problem persists. in the appsearch enterprise search (cloud) while running crawler. the amount… --- ## [How to index only given urls in the Elasticsearch using Open Crawler](https://discuss.elastic.co/t/how-to-index-only-given-urls-in-the-elasticsearch-using-open-crawler/361899)
**Author:** [@jahedi](https://discuss.elastic.co/u/jahedi)\ **Replies:** 3\ **Last updated:** [June 26, 2024, 8:45am UTC](https://discuss.elastic.co/t/how-to-index-only-given-urls-in-the-elasticsearch-using-open-crawler/361899 "2024-06-26T08:45:13Z")
Hi, I was just wondering how I can index only those urls that I have given in the crawlerconfig.yml file. Let me explain better, for example I have this config file for the crawler: domain\_allowlist: - https://a.com … --- ## [Web Crawler API](https://discuss.elastic.co/t/web-crawler-api/359385)
**Author:** [@Chenko](https://discuss.elastic.co/u/Chenko)\ **Replies:** 2\ **Last updated:** [June 20, 2024, 2:47pm UTC](https://discuss.elastic.co/t/web-crawler-api/359385 "2024-06-20T14:47:39Z")
Hi, Reposting this issue because it did not get answered yet. I got a couple of questions regarding the Web Crawlers API's (App Search's crawler & Elastic Web Crawler): Why does the more powerful web crawler (Elastic… --- ## [Apply delay between requests in Elastic Crawler](https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375)
**Author:** [@jahedi](https://discuss.elastic.co/u/jahedi)\ **Replies:** 3\ **Last updated:** [June 17, 2024, 7:23am UTC](https://discuss.elastic.co/t/apply-delay-between-requests-in-elastic-crawler/361375 "2024-06-17T07:23:42Z")
Hi, I am using the recently released project , Elastic Crawler, and I can not find any configurations for making some delay between each request while crawling on a domain. Is there any config to set a number to have a… --- ## [Supporting stop words in Elastic Open Crawler](https://discuss.elastic.co/t/supporting-stop-words-in-elastic-open-crawler/361532)
**Author:** [@jahedi](https://discuss.elastic.co/u/jahedi)\ **Replies:** 1\ **Last updated:** [June 17, 2024, 7:12am UTC](https://discuss.elastic.co/t/supporting-stop-words-in-elastic-open-crawler/361532 "2024-06-17T07:12:58Z")
Hi, I am testing the currently released project, Open Crawl, for a few days. I was just wondering if this project supports the stop words? In other word I don't want some words like "the", "and", "a", etc, effects on my…