Could not add the UUID fingerprint in producer on logstash

Hi Team,

Im adding the fingerprint of method UUID to avoid the duplication of the data.
Im using kafka as producer and Elasticsearch as consumer.

Example pipeline conf looks like this

                bootstrap_servers => ["xxxxxxx:9093"]
                topics => ["xxxxxx"]

filter {
  fingerprint {
    target => "generated_id"
    method => "UUID"

output {
      elasticsearch {
                hosts => ["https://xxxxxxxxxx:9200"]
                document_id => "%{[generated_id]}"

Using this method we are not able to achieve the deduplication of data on the Elasticsearch side.

Can we solve this problem using any other approach?

Yes, use a fingerprint filter but do not use UUID. See the documentation:

When the method option is set to UUID the result won’t be a consistent hash but a random UUID

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.