Duplicates in ES Index

Elastic Stack Logstash
1

My ELK Stack is running a process which collects logs from 5-6 different PCs.
The only issue I am facing at the moment is, my ES index is logging multiple entries for the same timestamp.

Any suggestions on how I can fix and remove the duplicates?

The following is an example of how the data is indexed. All 4 entries have the exact same data and I just need one of these. -> This affects the dashboards and the results that we're trying to observe.
image

2

You can add this in your output, and then you will have 'update' on doc, and no more duplicates.

document_id => "%{[field_with_same_value_for_duplicate_docs]}"

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.