I am using logstash pipeline data from elasticsearch to kafka. I have an index(asd_logstash-2018.07.15) with around 1000 docs, but upon checking in kafka it went around 22k messages and continuously increasing. I came across this blog https://www.elastic.co/blog/logstash-lessons-handling-duplicat…

Duplicated data in kafka using logstash

jogoinar10 (Jonar B) August 10, 2018, 6:35am 2

I tried to use fingerprint filter plugin from logstash but the output if kafka still duplicates.

Logstash kafka output plugin

Topic		Replies	Views
Logstash kafka output plugin Logstash	5	471	September 8, 2018
Duplicate records found Logstash	1	297	October 17, 2018
Logstash Kafka input causes duplicates Logstash	3	1386	September 12, 2017
Is it possible detect duplicate message in Kafka topic with logstash Logstash	2	466	August 28, 2019
Found duplicate records in elasticsearch Logstash	8	2487	December 25, 2017

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Duplicated data in kafka using logstash

Related topics