# File rotation duplicate data parsed by LS. duplicate data in ES

**URL:** <https://discuss.elastic.co/t/file-rotation-duplicate-data-parsed-by-ls-duplicate-data-in-es/98083>\
**Category:** Logstash\
**Created:** [August 23, 2017, 1:31pm UTC](https://discuss.elastic.co/t/file-rotation-duplicate-data-parsed-by-ls-duplicate-data-in-es/98083 "2017-08-23T13:31:16Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sezanawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sezanawa/32/21910_2.png) [@sezanawa](https://discuss.elastic.co/u/sezanawa)\
**Post date:** [August 23, 2017, 1:31pm UTC](https://discuss.elastic.co/t/file-rotation-duplicate-data-parsed-by-ls-duplicate-data-in-es/98083/1 "2017-08-23T13:31:16Z")

</div>

Hallo Eveybody

I am using ELK for Log parsing. I have a Job which dowload Logfiles after every 15 mins to a central log directory. Logstash is configured to parse log files from central directory and send to ES.

as i noticed logstash send same log lines after every 15 mins to ES. It mean after an hour i have same Error message with same timestampt etc 4 times in ES. I am using following configuration in file input filter of logstash.

```
file {
   	type => "OutputManagement"	
	path => ["D:/logs/ApplicationEntLib*.log"]
	start_position => "end" 
	#sincedb_path => "NUL" 
	ignore_older => 90000 		 	
	codec => multiline {
	  pattern => "^%{WORD};"
	  negate => true
	  what => "previous"
	}

```

I have following questions.

How do i tell logstash to only parse the new lines in a log file ?  
On Date Change at server my application which using Enterprise Library rename ApplicationEntLib.log Logfile to ApplicationEntLib-2017-08-23.log. will logstash parse this file again ?

Any idea how stop duplicate parsing of log files ?

Thanks in advance

best regards

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 25, 2017, 8:08am UTC](https://discuss.elastic.co/t/file-rotation-duplicate-data-parsed-by-ls-duplicate-data-in-es/98083/3 "2017-09-25T08:08:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.