Duplicate records in Kibana

[apurvakhatri]

I noticed that there are multiple records for the same log. They have the same clientip, time & host.

Issue: Duplicate records. (There are multiple copies of same record)

Requirement: There should only be one record of any particular log injected into elasticsearch.

Any Solutions would be of great help!

[Badger]

Build a fingerprint based on the fields that make the record unique and use it to set the document_id on the elasticisearch output.

[apurvakhatri]

Thanks! I'm using it. It still happens but for very few number of records.
I want to know but why?
Logstash should send only one record, right? Filebeat also maintains state of files by keeping a record of offset in the registry.

[apurvakhatri]

Thank you!
Can you tell me why it happens?
Logstash should send only one record, right? Filebeat also maintains state of files by keeping a record of offset in the registry.