# Duplicate log events with different \_id generated by logstash

**URL:** <https://discuss.elastic.co/t/duplicate-log-events-with-different-id-generated-by-logstash/227135>\
**Category:** Logstash\
**Created:** [April 8, 2020, 1:23pm UTC](https://discuss.elastic.co/t/duplicate-log-events-with-different-id-generated-by-logstash/227135 "2020-04-08T13:23:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nityaraj06](https://avatars.discourse-cdn.com/v4/letter/n/d78d45/32.png) [@nityaraj06](https://discuss.elastic.co/u/nityaraj06)\
**Post date:** [April 8, 2020, 1:23pm UTC](https://discuss.elastic.co/t/duplicate-log-events-with-different-id-generated-by-logstash/227135/1 "2020-04-08T13:23:40Z")

</div>

Hi,

I am able to see duplicate log events with different \_id generated by logstash in Kibana. Below is my logstash config. Could someone help me with this?

```auto
input {
    file {
		path => "E:/console-2020*.log"		
		sincedb_path => "E:\logstash\logstash-7.6.0\sincedb\console-sincedb.txt"
		type => "consolelogs"
		codec => multiline {
			pattern => "%{WORD}%{SPACE}\|%{SPACE}jvm%{SPACE}1%{SPACE}\|%{SPACE}srvmain%{SPACE}\|%{SPACE}%{YEAR}/%{MONTHNUM2}/%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}.%{SECOND}%{SPACE}\|%{SPACE}at"
			what => "previous"
		}
    }
}

filter {	
	if "DefaultSolrClientPool" in [message] {
		drop { }
	}

	grok {
        }
	
		}
	}
		
	date {
		match => ["newtimestamp", "yyyy/MM/dd HH:mm:ss.SSS"]
		timezone => "Europe/Stockholm"
		target => "@timestamp"
	}
}
			
output {
	if [type] == "consolelogs" {
		elasticsearch {
			hosts => ["<ip1>:9200"]
			index => "console-preprod-%{+YYYY.MM.dd}"
		}
	}
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 9, 2020, 5:29pm UTC](https://discuss.elastic.co/t/duplicate-log-events-with-different-id-generated-by-logstash/227135/2 "2020-04-09T17:29:34Z")

</div>

Is it reading from a local drive or is this networked storage? How is the log file updated/append to?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2020, 5:29pm UTC](https://discuss.elastic.co/t/duplicate-log-events-with-different-id-generated-by-logstash/227135/3 "2020-05-07T17:29:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.