Slow Ingestion of Final Log Chunks (Filebeat + Logstash + Elasticsearch)

[RainTown (Kevin Maguire)]

How to deal with duplicate data

I have a Filebeat pipeline that is ingesting data from an end-user machine that might be stored there for 30 days. My Logstash pipeline has the following settings: document_id => "%{[@metadata][newId]}" action => "create" Because I wanted to make sure that the same log is never written into the database twice I set the action to create and I created my unique document_id. That setup works fine for me but I had a situation recently where Filebeat was uninstalled and the registry folder for it w…

is the threads I was half-recalling, which was sort of similar, but the other way round (lots of errors on failed creates, rather than asking ES to do un-necessary updates).

Good luck.