Delete several documents in elastic with logstash?

[nicolasl]

Hello,

I would like to delete multiple document with logstash in elasticsearch before sending the data to elastic. Why .. ?

For example, I ingest a file into elastic with logstash. On the next day, I receive an updated version of that file. Some of the data inserted the first day were update, some added and some removed.

So I would like to delete all previously ingested data between 2 date and then ingest the new data. I have to do so because if I only update the data, my removed data in the file will not be removed in elastic.

I can delete 1 document and only 1 using the action delete in my elastic output. As I would like to delete multiple documents so it does not work for me.

The only way I see is to use a ruby script in the filter part and use _delete_by_query with curl but it is a bit killer. Maybe I missed a better way to do it ?

Or maybe with the http output pluging and the post method ?

Thanks.

[nicolasl]

Ok I have found a way with the http output pluging and the post method :

http {
          url => "http://es-svc:25000/logstash-test-xml-data-mp/_delete_by_query"
          http_method => "post"
          format => "message"
          content_type => "application/json"
          message => '{"query":{"range":{"observationtimestart":{"gte":"%{ValidityStart}","lte":"%{ValidityStop}"}}}}'
        }