Delete or update document without document_id

Hi, i'm new on ES, i'am using logstash and csv to update or delete values into ES.
I want to delete or update document based on different field not only with document_id

like this:

input {
  stdin{}
}

filter {
  csv {
      separator => ","
      skip_header => "true"
      columns => ["productCode","productName","productLine","operation"]
  }
  
}

output {
       if [operation] == "delete" {
        elasticsearch {
                    action => "delete" 
                    delete by product code
                    index => "product"
                    hosts  =>  ["127.0.0.1:9200"]
        }
        }if [operation] == "update" {
           elasticsearch {
                    action => "update" 
                   update by product code 
                    index => "product"
                    hosts  =>  ["127.0.0.1:9200"]
        }
        }
        stdout { codec => rubydebug }
}

In this case, product code is not the document_id. I don't have the document id in the csv, so i'am not able to find the specific document to delete or update, i have only product code or maybe other fields.
Can i do something like this?

The elasticsearch output in logstash can only delete by id. However, elasticsearch itself has delete-by-query and update-by-query APIs, which you may be able to use from a logstash http filter or http output plugin.

Can you give me an example?

There is an example of connecting to elasticsearch using an http filter here. It is an update rather than update-by-query, but the API documentation explains how to do the latter.

This is my filter and output code, but i have a problem to use "product code" inside the json

filter {
  csv {
      separator => ","
      skip_header => "true"
      columns => ["productCode","productName","productLine","operation"]
  }
  
}

output {
       if [operation] == "delete" {
       http {
        url => "http://localhost:9200/product/_delete_by_query"
        http_method => "post"
        body_format => "json"
        content_type => "application/json"
        body => '{
                "query": {
                    "match": {
                      "productCode": '[productCode]'
                              }
                        }
                    }'
      }
      }
      stdout { codec => rubydebug }
}

You should use a sprintf reference

"productCode": "%{productCode]}"

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.