Slow Ingestion of Final Log Chunks (Filebeat + Logstash + Elasticsearch)

[RainTown (Kevin Maguire)]

@RafaelXokito

The log files have duplicated logs, we receive a ton of log files from a third party and we need to process them, unfortunately the logs contain duplicates (a lot of duplicates). Meaning that, even with a single filebeat replica I have duplicates.

You could maybe have shared this a bit earlier in the thread .... ?

There's other ways you could de-duplicate your data before ingest, could even be done in logtsash itself, see

I Want to remove the duplicate events inside Logstash filter how could I do that? I mention the events below please have a look and suggest

You can use a fingerprint filter with the concatenate_all_fields option set to true. If you are sending events to elasticsearch then use the fingerprint as the document_id and duplicate events will be overwritten. If you really want to do the de-duplication in logstash (because you are not writing to elasticsearch) then you would need to use a ruby filter that builds a cache of recently seen fingerprints. You would look for the fingerprint in the cache and event.cancel if it is found, or add it…

But would add significant complexity.

Someone recently had similar issue and was using action => "create" the first time it saw docX, then getting (expected) errors on further create (not update) requests for same docX (same _id), and wanted to squash the error (or was it warning?) messages. This might be quicker.