Logstash Filters

Elastic Stack Logstash
1

Hi Good morning,

I have a requirement of creating a filter in logstash which is displayed as a field in kibana also.

if any (Exception or IOException or NullPointerException) in message, then create a field called msg_exception in logstash.

find my configuration:

filter{
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:time} [%{NUMBER:thread}] %{LOGLEVEL:loglevel} %{GREEDYDATA:class} -%{GREEDYDATA:msg} " }
}
if "Exception" in [msg] {
grok {
add_field => { "msg_error" => "%{msg}" }
}
}
}

Problem is data is loaded into ES through Logstash , but am not getting any filter.

2

When you have an add_field declaration in a filter it means "add the fields if the filter is successful", but in your second grok filter you're specifying any matches criteria so the filter probably doesn't consider itself successful. Use a mutate filter instead:

mutate {
  add_field => { "msg_error" => "%{msg}" }
}

Also, avoid using multiple GREEDYDATA patterns. In this case you're probably okay most of the time but it can really mess things up if one isn't careful. In this case I suggest using %{JAVACLASS:class} or %{NOTSPACE:class} instead (and consider naming the field e.g. logger since that describes better what the field actually contains).

3

@magnusbaeck, here am getting the problem like all records are displaying when click on msg_error,

But i need only where Exception| IOException| NullPointerException is matching that record need to be filtered in kibana.

Problem in Kibana:

4

You want to list only messages whose msg_error field is set? In that case, add e.g. AND msg_error:* to the query.

5

I am writing the filter like this,

filter{
grok {
match => { "message" =>
"%{TIMESTAMP_ISO8601:time} [%{NUMBER:thread}] %{LOGLEVEL:loglevel}
%{NOTSPACE:logger} -%{GREEDYDATA:msg} " }
}
if "Exception" in [msg] {
grok {
add_field => { "msg_error" => "%{msg}AND msg_error:*" }
}
}
}

Is it right ?? i am confusing where to write the query ...

6

The Kibana query. If, as I said, the problem is that seeing all kinds of events and not just those with the msg_error field set. Also, you're still using the grok filter even though I suggest that you use a mutate filter. That might work but it doesn't look right.

7

Ya right, it is working when the word is only Exception. but i need the solution for IOException and NullPointerException(search with the Exception, IOException and other exception are not searched)

Here i am getting am getting a new problem (-grokparsefailure) when this type of log is loading into ES

Actually my log is:

Pasted image1165×98 15.9 KB

Pattern is

"%{TIMESTAMP_ISO8601:time} [%{NUMBER:thread}] %{LOGLEVEL:loglevel}
%{NOTSPACE:logger} -%{GREEDYDATA:msg} " }

-> is there any need to add the new pattern in filter ??

existing filter configuration is:

filter{
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:time} [%{NUMBER:thread}] %{LOGLEVEL:loglevel} %{GREEDYDATA:class} -%{GREEDYDATA:msg} " }
}
if "Exception" in [msg] {
mutate {
add_field => { "msg_error" => "%{msg}" }
}
}
}

8

Ya right, it is working when the word is only Exception. but i need the solution for IOException and NullPointerException(search with the Exception, IOException and other exception are not searched)

Whether it's just Exception or IOException doesn't matter. if "Exception" in [msg] is a "stupid" substring search that doesn't care about word boundaries.

"%{TIMESTAMP_ISO8601:time} [%{NUMBER:thread}] %{LOGLEVEL:loglevel}
%{NOTSPACE:logger} -%{GREEDYDATA:msg} " }

-> is there any need to add the new pattern in filter ??

There's a trailing space in your pattern that shouldn't be there. Also, square brackets are metacharacters so [%{NUMBER:thread}] must be \[%{NUMBER:thread}\].

Next time please don't post screenshots. Use the copy/paste feature that I'm sure you're you're familiar with.