File rotation duplicate data parsed by LS. duplicate data in ES

Elastic Stack Logstash
1

Hallo Eveybody

I am using ELK for Log parsing. I have a Job which dowload Logfiles after every 15 mins to a central log directory. Logstash is configured to parse log files from central directory and send to ES.

as i noticed logstash send same log lines after every 15 mins to ES. It mean after an hour i have same Error message with same timestampt etc 4 times in ES. I am using following configuration in file input filter of logstash.

file {
   	type => "OutputManagement"	
	path => ["D:/logs/ApplicationEntLib*.log"]
	start_position => "end" 
	#sincedb_path => "NUL" 
	ignore_older => 90000 		 	
	codec => multiline {
	  pattern => "^%{WORD};"
	  negate => true
	  what => "previous"
	}

I have following questions.

How do i tell logstash to only parse the new lines in a log file ?
On Date Change at server my application which using Enterprise Library rename ApplicationEntLib.log Logfile to ApplicationEntLib-2017-08-23.log. will logstash parse this file again ?

Any idea how stop duplicate parsing of log files ?

Thanks in advance

best regards

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.