# Duplicate Entries of Log data

**URL:** https://discuss.elastic.co/t/duplicate-entries-of-log-data/98894
**Category:** Elasticsearch
**Created:** [August 30, 2017, 4:40pm UTC](https://discuss.elastic.co/t/duplicate-entries-of-log-data/98894 "2017-08-30T16:40:23Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![nktvys](https://avatars.discourse-cdn.com/v4/letter/n/a183cd/32.png) [@nktvys](https://discuss.elastic.co/u/nktvys)
#### Post date: [August 30, 2017, 4:40pm UTC](https://discuss.elastic.co/t/duplicate-entries-of-log-data/98894/1 "2017-08-30T16:40:24Z")

</div>

Hi,

I have multiple copies for a single entry of a log file in Elasticsearch. For eg. In my original log file, for a timestamp 12:55:03:234 there is a log entry X. In Elasticsearch for that timestamp, I have duplicates of X (only difference is \_id). Now this number of duplicates keeps varying for different log entries and there are entries for which no duplicates exist.  
Please help me understand why this is happening?  
How can I remove the duplicate log entries from elastic?

Thanks!

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 30, 2017, 4:46pm UTC](https://discuss.elastic.co/t/duplicate-entries-of-log-data/98894/2 "2017-08-30T16:46:14Z")

</div>

Have a look at [this blog post](https://www.elastic.co/blog/logstash-lessons-handling-duplicates). What does your ingest pipeline look like?

---

<div class="post-metadata">

### Author: ![gautamv](https://avatars.discourse-cdn.com/v4/letter/g/ecccb3/32.png) [@gautamv](https://discuss.elastic.co/u/gautamv)
#### Post date: [September 1, 2017, 8:34am UTC](https://discuss.elastic.co/t/duplicate-entries-of-log-data/98894/3 "2017-09-01T08:34:12Z")

</div>

Hi,

I too have the same issue. Please find below my logstash config code:

```
input {

	file {
		type => "SystemError"
		path => "/data/others/*"
		start_position => "beginning"
		sincedb_path => "/dev/null"

		codec => multiline {
			pattern => "^(\s | \[%{DATA}] %{BASE16NUM} (?<shortname>\b[A-Za-z0-9\$]{2,}\b)%{SPACE}%{WORD}%{SPACE}\tat)"
			what => "previous"
		}

   	}

}
		

filter {
	
	grok {
		match => { "message" => "\[%{DATA:timestamp}] %{BASE16NUM:threadID} (?<shortname>\b[A-Za-z0-9\$]{2,}\b)%{SPACE}%{WORD:loglevel}%{SPACE} %{GREEDYDATA:message}" }
		overwrite => ["message"]
  	}

	date {
		match => ["timestamp", "M/dd/yy HH:mm:ss:SSS zZZ"]
	}

  	fingerprint {
    		source => ["@timestamp", "threadID", "message"]
		concatenate_sources => true
    		target => "%{[@metadata][fingerprint]}"
    		method => "MURMUR3"
  	}
}
	

output {
#	stdout { codec => rubydebug }

	elasticsearch {
        	hosts => ["x.x.x.x"]
		user => 'elastic'
		password => 'xxxx'
		document_id => "%{[@metadata][fingerprint]}"
		index => "logstash-%{+YYYY.MM.dd}"
		manage_template => false
  	}

}

```

Please help. The code is not working.

Thanks

---

<div class="post-metadata">

### Author: ![nktvys](https://avatars.discourse-cdn.com/v4/letter/n/a183cd/32.png) [@nktvys](https://discuss.elastic.co/u/nktvys)
#### Post date: [September 1, 2017, 8:35am UTC](https://discuss.elastic.co/t/duplicate-entries-of-log-data/98894/4 "2017-09-01T08:35:46Z")

</div>

I will go through the blog post and let you know if the problem gets solved.

Thanks!

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 1, 2017, 8:37am UTC](https://discuss.elastic.co/t/duplicate-entries-of-log-data/98894/5 "2017-09-01T08:37:15Z")

</div>

> [@gautamv](#):
>
> start\_position =\> "beginning"  
> sincedb\_path =\> "/dev/null"

This means that files will be reprocessed every time you restart Logstash, so I am not surprised you are seeing duplicates.

---

<div class="post-metadata">

### Author: ![gautamv](https://avatars.discourse-cdn.com/v4/letter/g/ecccb3/32.png) [@gautamv](https://discuss.elastic.co/u/gautamv)
#### Post date: [September 1, 2017, 8:38am UTC](https://discuss.elastic.co/t/duplicate-entries-of-log-data/98894/6 "2017-09-01T08:38:36Z")

</div>

Hi,

This code is just for testing purposes. I have removed `sincedb_path => “/dev/null”` in my actual code.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 29, 2017, 8:38am UTC](https://discuss.elastic.co/t/duplicate-entries-of-log-data/98894/7 "2017-09-29T08:38:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.