Multiple Config Files - Duplicate data

Elastic Stack Logstash
1

I am new to logstash and filebeat. I am trying to set up multiple config files for my logstash instance.
Using filebeat to send data to logstash. Even if I have filters created for both the logstash config files, I am getting duplicate data.

Logstash config file - 1:

input {
  beats {
    port => 5045
  }
}

filter {
   if [fields][env] == "prod" {
     grok {   match => { "message" => "%{LOGLEVEL:loglevel}] %{GREEDYDATA:message}$" }
     overwrite => [ "message" ]
     }
   }
}

output {
  stdout {
    codec => rubydebug
  }

  elasticsearch {
    hosts => ["https://172.17.0.2:9200"]
    index => "logstash-myapp-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "password"
    ssl => true
    cacert => "/usr/share/logstash/certs/http_ca.crt"
  }
}

logstash config file-2

input {
  beats {
    port => 5044
  }
}

filter {
   if [fields][env] == "dev" {
     grok {   match => { "message" => "%{LOGLEVEL:loglevel}] %{GREEDYDATA:message}$" }
     overwrite => [ "message" ]
     }
   }
}

output {
  stdout {
    codec => rubydebug
  }

  elasticsearch {
    hosts => ["https://172.17.0.2:9200"]
    index => "logstash-myapp-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "password"
    ssl => true
    cacert => "/usr/share/logstash/certs/http_ca.crt"
  }
}

Logfile Content:

[INFO] First Line
[INFO] Second Line
[INFO] Third Line

Filebeat config:

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /root/data/logs/*.log
  fields:
    app: test
    env: dev

output.logstash:
  # The Logstash hosts
    hosts: ["172.17.0.4:5044"]

I know that even if we have multiple files for config, logstash processes each and every line of the data against all the filters present in all the config files. Hence we have put filters in each of the config files for "fields.env".
I am expecting 3 lines to be sent to Elasticsearch because "fields.env" is "dev", but it is sending 6 lines to Elasticsearch and duplicate data.
Pleas help.

2

The conditional only controls which grok filters are applied. All events are written to all outputs unconditionally.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.