Why does it have duplicate ID with normal logstash output and fingerprint filter?

Elastic Stack Logstash
1

Hi team, This is my logstash config to get data, and I use fingerprint as _id in order to prevent duplication from data. But it still has duplicated event as below. Any help?

filter {
        if [type] == "cve" {
        fingerprint {
                key => "123ABC"
                method => "SHA256"
                source => ["id"]
                target => "[fingerprint]"
        }
        }
}

elasticsearch {
        hosts => ["http://1x.1x.1x.2:9200"]
        index => "hc-%{+YYYY.MM.dd}"
        document_id => "%{fingerprint}"
  }

The first event
"
_id
6bacf9435a4a7161994323d09d72ec3640418b31fb7544e5e3786fb7805a6b1d
_index
hc
_type
_doc
author
CVEnew
fingerprint
6bacf9435a4a7161994323d09d72ec3640418b31fb7544e5e3786fb7805a6b1d
"

The second event
"
_id
6bacf9435a4a7161994323d09d72ec3640418b31fb7544e5e3786fb7805a6b1d
_index
hc
_type
_doc
author
CVEnew
fingerprint
6bacf9435a4a7161994323d09d72ec3640418b31fb7544e5e3786fb7805a6b1d
"

image
image728×81 4.19 KB

2

Can you post the original document?

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.