Hi all, we are looking a bit arround how we can delete crawled "pages" (documents) from an index when they are outdated or change their nodindex from INDEX to index or if they match any other metadata. We experimented with Ingest Pipelines but they do not have processors for deleting documents fro…

Delete (Elastic Web crawler) crawled Web pages - maybe per Pipeline

Sean_Story (Sean Story) December 5, 2023, 2:30pm 5

@sebastianboelling if a full crawl is run, pages that change to noindex should be deleted. See our discussion on another forum post: Does Elastic Web Crawler supports noindex and nofollow directive - #9 by sebastianboelling

Topic		Replies	Views
Does Elastic Web Crawler supports noindex and nofollow directive Elastic Search elastic-app-search	10	640	December 8, 2023
Calling a pipeline on deletion operations Elasticsearch	4	390	July 23, 2019
After I delete a doc, and then I search it, I got it. after a while, when I search it, I can't get it Elasticsearch	3	352	July 5, 2017
Watcher delete document action Elasticsearch elastic-stack-alerting	2	648	August 17, 2022
Dynamically cancel processing in pipeline script (painless) Elasticsearch	4	663	October 17, 2017