Could not add the UUID fingerprint in producer on logstash

Elastic Stack Logstash
1

Hi Team,

Im adding the fingerprint of method UUID to avoid the duplication of the data.
Im using kafka as producer and Elasticsearch as consumer.

Example pipeline conf looks like this

input
{
        kafka
         {
                bootstrap_servers => ["xxxxxxx:9093"]
                topics => ["xxxxxx"]
               ......
               .......
        }
}

filter {
  fingerprint {
    target => "generated_id"
    method => "UUID"
  }
}

output {
      elasticsearch {
                hosts => ["https://xxxxxxxxxx:9200"]
                document_id => "%{[generated_id]}"

Using this method we are not able to achieve the deduplication of data on the Elasticsearch side.

Can we solve this problem using any other approach?

2

Yes, use a fingerprint filter but do not use UUID. See the documentation:

When the method option is set to UUID the result won’t be a consistent hash but a random UUID

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.