Please validate mutiple GROK filter

Elastic Stack Logstash
1

I Mentioned logstash-simple.conf file. Please validate logstash-simple conf file, whether imultiple GROK filters in correct format or not ? In case if it is wrong please mention the correct format. Thanks in Advance.

input {
beats
{
port => "5044"
}
}
filter{
kv {
filed_split => ","
source => "msg"
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{DATA:loglevel} %{DATA:source} %{DATA:node} %{DATA:index} %{DATA:link} %{DATA:colon} %{DATA:url} %{DATA:index} %{DATA:session} %{DATA:colon} %{DATA:userid} %{NOTSPACE:userid}" }
}
grok {

normal chat message entry

match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{DATA:loglevel} %{DATA:source} %{DATA:node} %{DATA:index} %{DATA:link} %{DATA:colon} %{DATA:url} %{DATA:index} %{DATA:session} %{DATA:colon} %{DATA:userid} " }
}
grok {

normal chat message entry

match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{DATA:source} %{DATA:colon} %{DATA:colon} %{DATA:message} %{DATA:URL} %{NOTSPACE:Link}%{GREEDYDATA:message}" }
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{DATA:source} %{DATA:colon} %{GREEDYDATA:message}" }
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{GREEDYDATA:message}" }
}
grok {
match => { "message" => "%{GREEDYDATA:message}" }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}

2

Please don't start new threads that continue the same question (How to use multiple filters and multiple Grok filters).

What you have works but most of your messages will get tagged _grokparsefailure since all grok filters are run on each message and there's probably at least one that doesn't match the message. It's also very inefficient. Instead, list multiple expressions in a single grok filter. There's an example of this in the grok filter documentation.

3

Thanks for your reply .

When i given two grok filter, it is always taking the first grok filter only. Eventhough second filter is correct, the data also related to second grok filter format, but it is taking the first grok filter message only.

Why it is not taking the second grok filter, even though the data is related to that format?
Is there is any other way to mention the format correctly ?
Please look the given format below.

filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{DATA:source} %{DATA:colon} %{GREEDYDATA:message}" }
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{GREEDYDATA:message}" }
}
}

4

When i given two grok filter, it is always taking the first grok filter only.

Nope. Please show an example that includes

  • your complete configuration,
  • the input string, and
  • what Logstash produces (use a stdout { codec => rubydebug } output).

But why are you bothering with two grok filters? Join them into a single filter.

Also, don't mix DATA and GREEDYDATA like this. It's inefficient and could result in incorrect matches. Use more exact patterns instead.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.