Hi Everyone, For years I have been using my own document _id to de-duplicate my data. The principle is that I use a ruby filter to create a unique hash of the IP address and the SHA fingerprint, and use this as the document_id. When I try to add to the index it will only add NEW ip+sha data and so …

Over on a separate thread in Elasticsearch I discussed and implemented 2 instances per node in my cluster, along with many of the other changes discussed here. I also changed my logstash configuration so that all data is sent to a single ES instance, and that instance can then pass the data within …

Discuss the Elastic Stack

Using my own document_id - is there a faster way?

Elastic Stack Logstash

Christian_Dahlqvist (Christian Dahlqvist) November 22, 2017, 8:13am 2

What kind of data are you indexing?

Topic		Replies	Views
How to create my own document_id in logstash? Logstash	19	26765	November 4, 2022
Avoid duplication Logstash	13	5215	December 7, 2018
Logstash produces duplicates Logstash	3	1187	July 6, 2017
Logstash->Elasticsearch document deduplication efficiency & optimization Elasticsearch	3	823	December 12, 2020
Indexing-time document deduplication Elasticsearch	6	2593	July 6, 2017

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Using my own document_id - is there a faster way?

Related topics