Duplicates in ES Index

My ELK Stack is running a process which collects logs from 5-6 different PCs.
The only issue I am facing at the moment is, my ES index is logging multiple entries for the same timestamp.

Any suggestions on how I can fix and remove the duplicates?

The following is an example of how the data is indexed. All 4 entries have the exact same data and I just need one of these. -> This affects the dashboards and the results that we're trying to observe.

You can add this in your output, and then you will have 'update' on doc, and no more duplicates.

document_id => "%{[field_with_same_value_for_duplicate_docs]}"

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.