I would like to delete multiple document with logstash in elasticsearch before sending the data to elastic. Why .. ?
For example, I ingest a file into elastic with logstash. On the next day, I receive an updated version of that file. Some of the data inserted the first day were update, some added and some removed.
So I would like to delete all previously ingested data between 2 date and then ingest the new data. I have to do so because if I only update the data, my removed data in the file will not be removed in elastic.
I can delete 1 document and only 1 using the action delete in my elastic output. As I would like to delete multiple documents so it does not work for me.
The only way I see is to use a ruby script in the filter part and use _delete_by_query with curl but it is a bit killer. Maybe I missed a better way to do it ?
Or maybe with the http output pluging and the post method ?