Issue with Index filtering within Logstash

Elastic Stack Logstash
1

Hello, we are currently using 5.3 of Elastic stack. We have two different teams using logstash so we have two different logstash conf files. One file is using UDP input with port 4558 and the other is using beats input with port 7777. Both outputs are sending to ES and have completely two different indexes. It seems the one with the beats input data is getting sent using the other index also. So we have duplicate data, data under the wrong index and errors in the logs for logstash because it is trying to parse the data as a JSON but there is no json data active. Is there something that can be causing this?

Thanks,
Kenneth

2

logstash filters and outputs are applied to all the data that comes in so if you have an event coming in "event1" this event will go through all the filters and the outputs.

you can add tags|types at the input level and put IFs ont he begining of your output to filter where its applied

hope this helps :slight_smile:

what i have is something like this:

input {
  beats {
    port => XXX
  }
}

output {
 if [type] == "wineventlog" {
  elasticsearch {
    hosts => "XXXXXX:9200"
    manage_template => false
    index => "XXXXXX-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
 }
}
3

I do have that setup which is weird to me.

output {
if [type] == "app-logs" {
elasticsearch {
hosts => "es-hostname:9200"
manage_template => false
index => "app-filebeat-%{+YYYY.MM.dd}"
}
}

the UDP logstash file does not though as I am not sure if there is a way to set a document type for that?

4

so you only have beats data in the beats index ? because you are filtering it but how is your other output ?

the config you pasted will make sure only "app-logs" write to "app-filebeat-*" bu it wont stop "app-logs" from writing to other index :wink:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.