Delete document in elastic in ingest based on condition

Elastic Stack Elasticsearch
1

my requirement is that I need to drop the records which are matching a condition. I am able to delete the records once it is indexed. but i would like to delete the documents using the ingest pipeline or filebeat filter.

POST my_index/_delete_by_query
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "testid"
}
}
}
}
}

I need the script something like below to delete document based on a condition.

{ "description" : "testing",
"processors": [ {
{ "script": { "lang": "painless", "inline": "delete document here"}
}
}]
}

2

Filebeat and ingest pipeline are data ingestion components and they can't be used to delete documents.

Is this a case that your application logs when a document is deleted?

3

Using filebeat i was able to drop events based on filter conditions
https://www.elastic.co/guide/en/beats/filebeat/master/drop-event.html
is it possible to do the same for case where if a field does not exist, drop the event?

I understood that using ingest we can modify the documents. is there way to drop that doc?

4

It sadly does not exist yet in ingest. See

5

Thanks @dadoonet

I could not really understand the workaround given in the issue using the on_failure. how can i work around the issue?

is there no way to do it?

-Vivek

6

I don't recall a real workaround for now for the use case you mentioned. But may be @talevy has an idea?