Hello, So the goal is to import existing mysql table which has about 2 million records in the ES index. However, ES index after a while has much more data. I also try to generate unique sha1 fingerprint of each message and use it as document_id to avoid, duplicates. However, even trough original …

Logstash produces duplicates

Topic		Replies	Views
Duplicate Entries of Log data Elasticsearch	6	4802	September 29, 2017
Avoid duplication Logstash	13	5215	December 7, 2018
Duplicate logs Elasticsearch	14	6871	July 10, 2018
Found duplicate records in elasticsearch Logstash	8	2522	December 25, 2017
Duplicate documents after Logstash restart Logstash	1	734	October 26, 2017

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Logstash produces duplicates

Related topics