Delete all documents from specific ES index, that are not in the logstash feed

Elastic Stack Logstash
1

Hello,

I am writing an ES ecommerce product search.

I feed data to ES using filebeat and logstash like this:
- every 24h I will upload a json file containing all the products in the ES index (all products that I want to search)
- I will have 2 pipelines opened for this:
- one for UPSERT on uid. This works perfectly.
- one that should DELETE all documents from ES that are no longer in the json file.
- every 5 min I will upload a json file with the products that have been modified by the ecommerce platform). This will use the UPSERT pipeline from above.

Is there a way to do the deletion part from the logstash conf file?

2

I would suggest including a "last updated" field in the upsert, then deleting any documents with an out of date "last updated" field.

3

Thank you for taking the time to help me with this. The "last update" suggestion is better than what I have right now.