Duplicate content in indexes

Elastic Stack Logstash
1

Hi,
I was hoping someone might be able to help me with this logstash.conf. Everything from beats is ending up in the wazuh index and the winlogbeat index. However the wazuh data does not end up in the winglogbeat index. If I comment out the beats input, I just get the wazuh data.

Many Thanks!

`input {
   file {
       type => "wazuh-alerts"
       path => "/var/ossec/logs/alerts/alerts.json"
       codec => "json"
   }
   beats {
       port => 5044
       ssl => true
       ssl_certificate => "/etc/pki/tls/certs/logstash.cer"
       ssl_key => "/etc/pki/tls/private/logstash.key"
  }

}

filter {
    if [type] == "wazuh-alerts" {
      if [data][srcip] {
          mutate {
              add_field => [ "@src_ip", "%{[data][srcip]}" ]
          }
      }
      if [data][aws][sourceIPAddress] {
          mutate {
              add_field => [ "@src_ip", "%{[data][aws][sourceIPAddress]}" ]
          }
      }
    }
}

filter {
    if [type] == "wazuh-alerts" {
      geoip {
          source => "@src_ip"
          target => "GeoLocation"
          fields => ["city_name", "continent_code", "country_code2", "country_name", "region_name", "location"]
      }
      date {
          match => ["timestamp", "ISO8601"]
          target => "@timestamp"
      }
      mutate {
          remove_field => [ "timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type","@src_ip"]
      }
    }
}



output {
  if [type] == "wazuh-alerts" {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
        document_type => "wazuh"
    }
  }
  else {
    elasticsearch {
      hosts => ["localhost:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  }
}

`

2

Is it possible that you are pointing -f to a directory that contains more than one configuration file (e.g. the one you want and a backup copy that has another output)? How are you starting logstash?

3

No it is just the logstash.conf. Also it stops if I comment the beats line out.

4

It's really hard to believe the events are flowing through both the if and else code blocks. When the beats events are in the wazuh index what document type do they have?

5

I had this issue myself. Turns out, there was an error in the Kibana index in Elasticseaerch. I got rid of that Kibana index and data was going to the correct indices.

Make sure the only files in /logstash/logstash.conf directory are only .conf files for logstash as well.

See here:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.