Two or more filters in Logstash

Elastic Stack Logstash
1

Hi folks!
May be somebody know how to use 2 or more filters block or 2 or more grok blocks in logstash configuration? My current configuration looks like this:

input {
udp {
port => 5514
type => syslog
}

}
input {
beats {
port => 5044
}
}
filter {
grok {
patterns_dir => ["/etc/logstash/patterns/ciscotest1"]
match => { "message" => "%{CISCONEWWIFIACCESSCREATED}"}
add_field => {
"Event_ID" => "CI-0001"
"Severity" => "30"
}
}
}
filter {
grok {
patterns_dir => ["/etc/logstash/patterns/ciscotest2"]
match => { "message" => "%{CISCOOLDCONNECTNOTVERIFYED}"}
add_field => {
"Event_ID" => "CI-0002"
"Severity" => "35"
}
}

}
output {
if "_grokparsefailure" in [tags] {
file {
path => "/var/log/custom/custom.log"
}
}
else {
elasticsearch {
hosts => ["testbuild12.local:9200"]
index => "index-%{+YYYY.MM.dd}"
}
}
}

So, it start without any problem, but marks all logs as "grokparsefailure".
In time when I use only one filter block for only one condition of grok - it works.

So the main question is, how to rebuild this config?

2

Hi

You only need one instance of each input{}, filter{} and output{}, and you can have as many plugins as you need in each of them.

In your case, you'll have one filter{} with two grok{} instances.

If you need one grok{} for your udp{} input and one for your beats{} input, you have to use tags and id to separate them.

Hope this helps

3

Hi!
So, I tried this configuration before with 2 grok elements in 1 filter - this cfg doesn't give any errors, but it just doesn't parse any logs.

4

Hi

You could try checking your match syntax, one grok{} at a time, against a known, simple, input. Send the output to stdout{} to see what you get, in detail. That will help you debug your code.

Hope this helps

5

Also tried.
Same result as in previous message.

6

I also had an idea of writing if statement and use regex for it like: "if [message] =~ /regex/" but it always give an error.

7

Hi

I think your match pattern should be something like

match => { "message" => "%{CISCONEWWIFIACCESSCREATED:cnwaccesscreated}"}

where cnwaccesscreated (i made the name up) is a new custom field containing the parsed contents from your message.

Hope this helps

8

So okay, I solved it.

More than 2 grok definitions can work only with if/elseif/else statement between them.
Without some type of stopper - it will grok ur log in every grok definition and send it to _grokparsefailure.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.