I am writing an ES ecommerce product search.
I feed data to ES using filebeat and logstash like this:
- every 24h I will upload a json file containing all the products in the ES index (all products that I want to search)
- I will have 2 pipelines opened for this:
- one for UPSERT on uid. This works perfectly.
- one that should DELETE all documents from ES that are no longer in the json file.
- every 5 min I will upload a json file with the products that have been modified by the ecommerce platform). This will use the UPSERT pipeline from above.
Is there a way to do the deletion part from the logstash conf file?