Logstash resending data after log file finishes

Elastic Stack Logstash
1

Hi I am monitoring a set of log files with Logstash and sending them to Elasticsearch.

But it seems like after the logs are done logstash resends the logs from the files again, leaving me with duplicate data in elasticsearch.

Here is my input :

    input{
     file {
     path => "/opt/hp1/bin/*pros*.log"
     type => "proslog"
     }

    file {
    path => "/opt/hp2/bin/*pros*.log"
    type => "proslog2"
    }
   }

my filter :

filter{
 if [type] == "proslog" or [type] == "proslog2" {
 grok {
 match => ["message", "(?<day>%{MONTHNUM}.%{MONTHDAY}.%{YEAR})%{SPACE}?%{TIME:timej}%{SPACE}product: SW"]
      }

 mutate {
 add_field => {
   "OriginalLogTime" => "%{day} %{timej}"
  }
 }

  date {
  match => [ "OriginalLogTime", "MM.dd.YY HH:mm:ss.SS"]
  target => "@timestamp"
  }
 }
}

and my output:

output{

 if [type] == "proslog" or [type] == "proslog2" {
   elasticsearch {
        index => "proslogs-%{+YYYY.MM.dd}"
        hosts => "http://production.bs.com:9200"
        manage_template => false
   }
 }
}

An example of what I am talking about is this, from kibana:

June 26th 2018, 05:23:24.918    06.26.18  5:22:39.56 product: SW num:155577wst

June 26th 2018, 06:27:44.380	06.26.18  5:22:39.56 product: SW num:155577wst

I get the exact same data about an hour later (the time the log files are finished). How can I get logstash not to resend the data from the logs after they finish?

*also the names of the files do not change after they finish. Not sure what is going on

thank you.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.