We're sending logs using filebeats to Logstash (Parsing with configs) --> LumberJack --> ES. In some situations while collecting newer data types, we want to re-index data,
so we do the following -
- stop filebeats
- stop logstash
- Change any logstash filters
- Delete Filebeat registry to re-read data
- start Logstash
- Start Filebeat
But during the step 5, when logstash is started, I think it tries to resend any inflight events it maybe holding up in the memory and creates the indexes with stale data in ES, but I want to prevent that from happening in this situation. I haven't been able to figure out any settings related that would help me preventing transmission of these events, as I want to re-index data.
Any suggestions or directions will be really helpful.