# Slow Ingestion of Final Log Chunks (Filebeat + Logstash + Elasticsearch) **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-monitoring\ **Created:** [April 16, 2025, 5:14pm UTC](https://discuss.elastic.co/t/slow-ingestion-of-final-log-chunks-filebeat-logstash-elasticsearch/377214 "2025-04-16T17:14:10Z")\ **Posts on this page:** 1\ **Showing post:** 16
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [April 18, 2025, 1:52pm UTC](https://discuss.elastic.co/t/slow-ingestion-of-final-log-chunks-filebeat-logstash-elasticsearch/377214/16 "2025-04-18T13:52:29Z")
> [@How to deal with duplicate data](https://discuss.elastic.co/t/how-to-deal-with-duplicate-data/375229): > > I have a Filebeat pipeline that is ingesting data from an end-user machine that might be stored there for 30 days. My Logstash pipeline has the following settings: document\_id =\> "%{[@metadata][newId]}" action =\> "create" Because I wanted to make sure that the same log is never written into the database twice I set the action to create and I created my unique document\_id. That setup works fine for me but I had a situation recently where Filebeat was uninstalled and the registry folder for it w… is the threads I was half-recalling, which was sort of similar, but the other way round (lots of errors on failed creates, rather than asking ES to do un-necessary updates). Good luck. --- _[View the full topic](https://discuss.elastic.co/t/slow-ingestion-of-final-log-chunks-filebeat-logstash-elasticsearch/377214)._