# Duplicate events even after introducing fingerprint

**URL:** <https://discuss.elastic.co/t/duplicate-events-even-after-introducing-fingerprint/256355>\
**Category:** Logstash\
**Created:** [November 23, 2020, 12:48pm UTC](https://discuss.elastic.co/t/duplicate-events-even-after-introducing-fingerprint/256355 "2020-11-23T12:48:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saravana37](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saravana37/32/85237_2.png) [@Saravana37](https://discuss.elastic.co/u/Saravana37)\
**Post date:** [November 23, 2020, 12:48pm UTC](https://discuss.elastic.co/t/duplicate-events-even-after-introducing-fingerprint/256355/1 "2020-11-23T12:48:57Z")

</div>

Hello All ,

We have a weird issue in ELK . We have introduced fingerprint in logstash , But still we can see the duplicates in Kibana. Kindly help how can we improve this .  
input {

```
	beats {
		port => 5044
		tags => ["srvr_logs"]
	}

}

filter  
{

if "metric" not in [fields][type]
{
	mutate {
			remove_field => ["host"]
		   }

		if "SOAPRouter" in [source]
		 {
		ruby {
		code => "
            event.set('Server', event.get('source').split('\\')[4])

        "
		}

		mutate {
			add_field => { 
				"Component" => "SOAPRouter"
            }
		}
grok {
patterns_dir => ["E:\ELK\logstash\patterns\patterns.txt"]
match => ["message", "%{SOAPROUTER}"]
}
}

else if "RequestRouter" in [source]
{

ruby {
        code => "
            event.set('Server', event.get('source').split('\\')[4])
            event.set('InterfaceName', event.get('source').split('\\')[5])

        "
    }
mutate {

add_field => { 

		"Component" => "RequestRouter"
               }
gsub => ["InterfaceName", "RequestRouter_", ""]            
}
ruby {
        code => "
             event.set('InterfaceName', event.get('InterfaceName').split('.')[0])

        "
    }
    
grok {
patterns_dir => ["E:\ELK\logstash\patterns\patterns.txt"]
match => ["message", "%{REQUESTROUTER1}",
"message", "%{REQUESTROUTER2}"]
}
     
mutate
{
gsub => ["EventType", "[0-9]", ""]
gsub => ["EventType", "\[\]", ""]
remove_field => ["NR1"]
remove_field => ["NR2"]
}
ruby {
        code => "
             event.set('EventType', event.get('EventType').split(' ')[0])

        "
    }

}
else if "eappweb" not in [source] and "RequestRouter" not in [source] and "SOAPRouter" not in [source]
{
ruby {
        code => "
            event.set('Server', event.get('source').split('\\')[4])
            event.set('Component', event.get('source').split('\\')[5])
        "
    }
mutate 
{
gsub => ["Component", "_[0-9].*", ""]
}

if "SBL" in [message]
{
grok {
patterns_dir => ["E:\ELK\logstash\patterns\patterns.txt"]
match => ["message", "%{SERVERLOG2}"]
}
}

else 
{
grok {
patterns_dir => ["E:\ELK\logstash\patterns\patterns.txt"]
match => ["message", "%{SERVERLOG1}"]
}
}
if "srvr_logs" in [tags] and "_grokparsefailure" in [tags]
{
grok {
patterns_dir => ["E:\ELK\logstash\patterns\patterns.txt"]
match => ["message", "%{SERVERLOG1}"]
}
}
}
else if "eappweb" in [source] 
{
 
mutate {

 rename => {
                        "[beat][hostname]" => "Server"
                }
add_field => { 

		"Component" => "eappweb"
               }
}

grok {
patterns_dir => ["E:\ELK\logstash\patterns\patterns.txt"]
match => ["message", "%{SERVERLOG1}"]
}

if "srvr_logs" in [tags] and "_grokparsefailure" in [tags]
{
grok {
patterns_dir => ["E:\ELK\logstash\patterns\patterns.txt"]
match => ["message", "%{SERVERLOG1}"]
}
}
}

if "RequestRouter" in [source] or "SOAPRouter" in [source]

{
date {
    			target => "EventDate"
 			match => ["EventDate", "yyyy-MM-dd HH:mm:ss,SSS"]
     }
}
else 

{
date {
    			target => "EventDate"
 			match => ["EventDate", "yyyy-MM-dd HH:mm:ss"]
     }
}

	fingerprint {
		source => ["message","source"]
		target => "[fingerprint]"
		key => "Compass"
		method => "MD5"
		concatenate_sources => true
	}

}
}

output {
	if "PROD" in [fields][env] and "metric" not in [fields][type]{

        elasticsearch {
            hosts => ["https:XXXXXXXXXXX:8200"]
			index => "prodsrvrlog-%{+YYYY.MM.dd}"
			user => "${es_usr}"
			password => "${es_pwd}"
			ssl => true
            ssl_certificate_verification => false
            cacert => "E:\ELK\ODForESearch\config\chain.pem"
            ilm_enabled => false
		}
		}
		
   else if "metric" in [fields][type]{

        elasticsearch {
            hosts => ["https:XXXXXXXXXX8200"]
			index => "metricbeat-%{+YYYY.MM.dd}"
			user => "${es_usr}"
			password => "${es_pwd}"
			ssl => true
            ssl_certificate_verification => false
            cacert => "E:\ELK\ODForESearch\config\chain.pem"
            ilm_enabled => false
		}
		
	}
}
```

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [November 23, 2020, 12:55pm UTC](https://discuss.elastic.co/t/duplicate-events-even-after-introducing-fingerprint/256355/2 "2020-11-23T12:55:57Z")

</div>

Hello Saravana,

I think you are missing a last step for it to work as you expect: The fingerprint filter does not remove duplicates - it just generates a unique ID which you can use to recognise duplicates.

To remove duplicates you have to configure you elasticsearch output to explicitly use the fingerprint value as [document id](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id). This way, the document will be created if a document with this fingerprint does not exist, otherwise the document will be overwritten.

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![Saravana37](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saravana37/32/85237_2.png) [@Saravana37](https://discuss.elastic.co/u/Saravana37)\
**Post date:** [November 23, 2020, 1:09pm UTC](https://discuss.elastic.co/t/duplicate-events-even-after-introducing-fingerprint/256355/3 "2020-11-23T13:09:03Z")

</div>

Hello @Wolfram_Haussig ,

I was in an assumption that fingerprint alone can remove duplicates .Thank you so much for letting me know about this.

I have added a piece of code to the output section . Can you please check if this is sufficient ?

```
elasticsearch {
    hosts => ["https:XXXXXXXXXXX:8200"]
		index => "prodsrvrlog-%{+YYYY.MM.dd}"
		user => "${es_usr}"
		password => "${es_pwd}"
		ssl => true
    ssl_certificate_verification => false
    cacert => "E:\ELK\ODForESearch\config\chain.pem"
    ilm_enabled => false
		document_id => "%{fingerprint}"
	}
```

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [November 24, 2020, 6:39am UTC](https://discuss.elastic.co/t/duplicate-events-even-after-introducing-fingerprint/256355/4 "2020-11-24T06:39:24Z")

</div>

Hello @Saravana37,

I think this should be sufficient. I am just not sure if it works with `document_id => "%{fingerprint}"`or if it has to be `document_id => "%{[fingerprint]}"`.

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![Saravana37](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saravana37/32/85237_2.png) [@Saravana37](https://discuss.elastic.co/u/Saravana37)\
**Post date:** [December 4, 2020, 4:53am UTC](https://discuss.elastic.co/t/duplicate-events-even-after-introducing-fingerprint/256355/5 "2020-12-04T04:53:33Z")

</div>

Thank you @Wolfram_Haussig ,

You are right .

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 1, 2021, 4:53am UTC](https://discuss.elastic.co/t/duplicate-events-even-after-introducing-fingerprint/256355/6 "2021-01-01T04:53:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.