Hi all, Logstash & Elasticsearch version - 5.4.3 Steps to produce the problem :- Use the below logstash.conf file input { file { path => ["path of the file"] start_position => "beginning" ignore_older => 0 } } output { elasticsearch { hosts => "10.0.X.X" manage_template => false inde…

Thanks @Christian_Dahlqvist for the replies. Fingerprint filter solved my problem. Regards Nikhil Kapoor

How to avoid data duplication in elasticsearch when data send from logstash?

Christian_Dahlqvist (Christian Dahlqvist) September 19, 2018, 5:26am 2

Have a look at this blog post, which covers how to avoid duplicates.

Topic		Replies	Views
How can i send duplicate lines (from logstash to elasticsearch) in log file Logstash	6	1165	April 13, 2020
Logstash write data to the elasticsearch how to remove duplication Logstash	4	678	July 6, 2017
How to avoid elasticsearch duplicate documents Logstash	6	1723	March 5, 2018
Duplicate logs Elasticsearch	14	6933	July 10, 2018
How to ignore the same log contents in the different log files? Logstash	5	459	June 13, 2018