# Delete (Elastic Web crawler) crawled Web pages - maybe per Pipeline

**URL:** <https://discuss.elastic.co/t/delete-elastic-web-crawler-crawled-web-pages-maybe-per-pipeline/344974>\
**Category:** Elastic Search\
**Tags:** painless, elastic-app-search\
**Created:** [October 13, 2023, 8:36am UTC](https://discuss.elastic.co/t/delete-elastic-web-crawler-crawled-web-pages-maybe-per-pipeline/344974 "2023-10-13T08:36:01Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sebastianboelling](https://avatars.discourse-cdn.com/v4/letter/s/258eb7/32.png) [@sebastianboelling](https://discuss.elastic.co/u/sebastianboelling)\
**Post date:** [October 13, 2023, 8:36am UTC](https://discuss.elastic.co/t/delete-elastic-web-crawler-crawled-web-pages-maybe-per-pipeline/344974/1 "2023-10-13T08:36:01Z")

</div>

Hi all,

we are looking a bit arround how we can delete crawled "pages" (documents) from an index when they are outdated or change their nodindex from INDEX to index or if they match any other metadata.

We experimented with Ingest Pipelines but they do not have processors for deleting documents from an index (of course they are about "Ingest"). Also Painless do not have functions for deleting documents based on their metadata, maybe if the puplishing date is older than a specified date.

What could be an approach to run some scheduled "cleaning" process of documents in an index which are not deleted by the web crawler because no 4xx or 5xx is thrown.

Of course we could run manually delete\_by\_query, but we are looking for a bit more automation or integration in pipelines.

Best regards

Sebastian

---

<div class="post-metadata">

**Author:** ![video](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/video/32/117114_2.png) [@video](https://discuss.elastic.co/u/video)\
**Post date:** [October 13, 2023, 4:18pm UTC](https://discuss.elastic.co/t/delete-elastic-web-crawler-crawled-web-pages-maybe-per-pipeline/344974/2 "2023-10-13T16:18:50Z")

</div>

Hi Sebastian,

There is no dedicated "cleaning" process, however, Elastic Web Crawler deletes pages that have \>= 300 && \<= 599 response codes during the full crawl.

Crawler should also remove pages from the index that have "noindex" tag but I need to double-check this. I'll keep you posted.

Does your configured Crawler delete pages during full crawls? If not, could you please provide some Crawler logs that include not deleted pages?

---

<div class="post-metadata">

**Author:** ![sebastianboelling](https://avatars.discourse-cdn.com/v4/letter/s/258eb7/32.png) [@sebastianboelling](https://discuss.elastic.co/u/sebastianboelling)\
**Post date:** [November 6, 2023, 1:32pm UTC](https://discuss.elastic.co/t/delete-elastic-web-crawler-crawled-web-pages-maybe-per-pipeline/344974/3 "2023-11-06T13:32:01Z")

</div>

Hi video,

sorry my first sentence was a bit weird. What I meant was:

1. A page was crawled indexed with `<meta name= ‘‘robots“ content= ‘‘index“>`
2. A page is re-crawled with `<meta name= ‘‘robots“ content= ‘‘noindex“>`

Will the page be deleted from index?

Did you have the opportunity to check it out?

Regards

Sebastian

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 4, 2023, 1:32pm UTC](https://discuss.elastic.co/t/delete-elastic-web-crawler-crawled-web-pages-maybe-per-pipeline/344974/4 "2023-12-04T13:32:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![Sean\_Story](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sean_story/32/69987_2.png) [@Sean\_Story](https://discuss.elastic.co/u/Sean_Story)\
**Post date:** [December 5, 2023, 2:30pm UTC](https://discuss.elastic.co/t/delete-elastic-web-crawler-crawled-web-pages-maybe-per-pipeline/344974/5 "2023-12-05T14:30:36Z")

</div>

@sebastianboelling if a full crawl is run, pages that change to `noindex` should be deleted. See our discussion on another forum post: [Does Elastic Web Crawler supports noindex and nofollow directive - #9 by sebastianboelling](https://discuss.elastic.co/t/does-elastic-web-crawler-supports-noindex-and-nofollow-directive/344821/9)