Several config files for beats in logstash

Elastic Stack Logstash
1

Hi,
We just started to work with logstash recently and i have a few questions about beats config files.

We currently have 2 configuration files on our logstash - one for filebeat and another one for winlogbeat .
Each config file configured to listen on different port (5044/5045) and configured to write to different index in elasticsearch.
It seems that every message is being processed by these two config files and all the messages, both from winlogbeat and filebeat, are being written to these 2 indices.

should it work this way? When logstash gets messages from beats application it's processing it in all the config files with beats input logstash has?

Our logstash version is 2.3.1 and these are the config files:

config1(winglogbeat):

input {
beats {
port => 5044
}
}

filter {
if [type] == "wineventlog" and [source_name] != "Test" {
drop { }
}
}

output {
elasticsearch {
hosts => "SERVER_IP"
manage_template => false
index => "winlogbeat-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

config2(filesbeat):
input {
beats {
port => 5045
}
}

filter {
grok {
match => { "message" => "%{DATESTAMP:date} %{WORD:timezone}] %{GREEDYDATA:data}" }
}
date {
match => [ "date", "MM/dd/YY HH:mm:ss:SSS" ]
}
}

output {
elasticsearch {
hosts => "SERVER_IP"
manage_template => false
index => "filebeat-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Thanks a lot!

2

This is expected. Logstash has a single event pipeline even if you split your configuration into multiple files. If you don't want all filters and output to apply to all events I suggest you use conditionals (e.g. based on the type field).

3

OK.
So, it should work as i expect if i'll delete output section on the files i mentioned and create new config file named beats_output.conf with this:

output {
if [type] == "wineventlog"
{
elasticsearch {
hosts => "SERVER_IP"
manage_template => false
index => "winlogbeat-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

if [type] == "OTHERFILEBEAT"
{
elasticsearch {
hosts => "SERVER_IP"
manage_template => false
index => "filebeat-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}

correct?

4

Yes, although you probably want to use the same conditionals for your filters.

5

Thanks! it works :slight_smile: