Handling multiple tags and types in LS configuration file

Elastic Stack Logstash
1

Hello, I would like to use my existing working LS config file to handle other log sources but I'm not sure of the syntax to handle multiple tags and types. I tried to the second config file below but it did not work. Any ideas? Thanks in advance.

existing working conf file:

input {
beats {
port => "5043"
tags => ["aemlogs","fglam"]
type => "linux-logs"
}
}
output {
if ("aemlogs" in [tags]) {
redis {
data_type => "list"
key => "linux-beats"
congestion_threshold => "2200000"
}
}
}

second NOT working conf file:

input {
beats {
port => "5043"
tags => ["aemlogs","fglam"]
type => ["linux-logs","fglam_beats"]
}
}
output {
if ("aemlogs", "fglam" in [tags]) {
redis {
data_type => "list"
key => "linux-beats"
congestion_threshold => "2200000"
}
}
}

2

The conditional syntax would be

    if "foo" in [tags] or "bar" in [tags] {
        mutate { add_tag => [ "atLeastOne" ] }
    }
    if "foo" in [tags] and "bar" in [tags] {
        mutate { add_tag => [ "both" ] }
    }

Are you using pipelines? If not, and -f points to a directory, logstash will concatenate all the files in the directory to create the configuration. That means you cannot have two beats inputs on the same port number -- one will get an address already in use.

3

[quote="Badger, post:2, topic:167904"]
Thanks Badger, I'm not sure if I'm using a pipelines, although I do get that error message if I try to use the same port number on multiple inputs. this is my running command:

logstash 16967 1 9 10:58 ? 00:00:50 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash

Is it recommended to have a different output filter for each tag? for example

output {
if "foo" in [tags] {
redis {
data_type => "list"
key => "linux-beats"
congestion_threshold => "2200000"
}
}

if "bar" in [tags] {
redis {
data_type => "list"
key => "linux-beats"
congestion_threshold => "2200000"
}
}

4

OK so you are not using pipelines and it is concatenating the configuration files.

If the two outputs are identical I would use a single output. But what you show is how to do conditionals in output sections.

5

I suppose what I'm trying to do is use port 5043 for all linux filebeat logs but keep separate tags. Is that recommended or does it not matter?

6

It matters if you need to conditional filters or outputs. If you have Windows event logs coming from one place, and syslog files from another and tomcat logs from somewhere else then tagging them at the source (i.e. filebeat) makes sense. You can then use the tags to put each set of logs through different filters.

7

Thanks @Badger That's what I thought. Much appreciated.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.