# Slow Ingestion of Final Log Chunks (Filebeat + Logstash + Elasticsearch)

**URL:** https://discuss.elastic.co/t/slow-ingestion-of-final-log-chunks-filebeat-logstash-elasticsearch/377214
**Category:** Elasticsearch
**Tags:** elastic-stack-monitoring
**Created:** [April 16, 2025, 5:14pm UTC](https://discuss.elastic.co/t/slow-ingestion-of-final-log-chunks-filebeat-logstash-elasticsearch/377214 "2025-04-16T17:14:10Z")
**Posts on this page:** 1
**Showing post:** 14

<div class="post-metadata">

### Author: ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)
#### Post date: [April 18, 2025, 9:42am UTC](https://discuss.elastic.co/t/slow-ingestion-of-final-log-chunks-filebeat-logstash-elasticsearch/377214/14 "2025-04-18T09:42:25Z")

</div>

> [@RafaelXokito](#):
>
> The log files have duplicated logs, we receive a ton of log files from a third party and we need to process them, unfortunately the logs contain duplicates (a lot of duplicates). Meaning that, even with a single filebeat replica I have duplicates.

You could _maybe_ have shared this a bit earlier in the thread .... ?

There's other ways you could de-duplicate your data before ingest, could even be done in logtsash itself, see

> [@I Want to remove the duplicate events inside Logstash filter how could I do that? I mention the events below please have a look and suggest](https://discuss.elastic.co/t/i-want-to-remove-the-duplicate-events-inside-logstash-filter-how-could-i-do-that-i-mention-the-events-below-please-have-a-look-and-suggest/349175/2):
>
> You can use a fingerprint filter with the [concatenate\_all\_fields](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html#plugins-filters-fingerprint-concatenate_all_fields) option set to true. If you are sending events to elasticsearch then use the fingerprint as the document\_id and duplicate events will be overwritten. If you really want to do the de-duplication in logstash (because you are not writing to elasticsearch) then you would need to use a ruby filter that builds a cache of recently seen fingerprints. You would look for the fingerprint in the cache and event.cancel if it is found, or add it…

But would add significant complexity.

Someone recently had similar issue and was using `action => "create"` the first time it saw docX, then getting (expected) errors on further create (not update) requests for same docX (same \_id), and wanted to squash the error (or was it warning?) messages. This might be quicker.

---

_[View the full topic](https://discuss.elastic.co/t/slow-ingestion-of-final-log-chunks-filebeat-logstash-elasticsearch/377214)._