PipeLine-> Filters-> (tags?)-> Is it possible to parse message by specific keywords after pattern parsing log message

[DanaDascalu]

Hi,

I new to using ELK stack, and need to create dashboards of meaningful data from our log files.
Currently i'm using filebeat to read log files, then parsing the messages using LogStash, and later on view the data in Kibana.

My logstash filter works for parsing the log messages to fields by matching a pattern, in the following way:
filter{
grok {
match => [
"message", "%{DATESTAMP:LogDate} [%{WORD:LoggerName}] [%{NUMBER:ThreadNumber}] -%{LOGLEVEL:Level} - %{DATA:LogText}$"
]
}
}

What I want to do next is parse the LogText field I created by specific keywords, so I can later on use them to display the data in Kibana dashboard.

for example:
2018-12-18 15:21:52,414 [MyLogger] [139] INFO - Text of the log message containing specific keywords.

The raw message is parsed as expected, and I get the LogText field which I want to further parse. In this exmple, I would like to eventually have a field or a tag called/containing "specific keywords".
How can I do this, if at all possible?
I tried adding a second filter that adds a tag, but failed, all my exploring only lead to log stash error while loading the process.

[Badger]

You can certainly parse LogText. There are several ways to do this. A grok or dissect filter might be applicable. Or regexp matching in a conditional, such as

if [LogText] =~ "specific keywords" {

Or substring matching in a conditional such as

if "specific" in [LogText] {

What did you try, and what was the resulting error?

[DanaDascalu]

Thanks for your reply.

I tried adding tags within and outside the previous filter. and got an error starting the logstash process, of unexcpected syntax.

Please help further:
should the if you suggested be inside the pattern filter? and what is the rest of the syntax to create a field out of the specific keyword?

I tried this:
filter{
grok {
match => [
"message", "%{DATESTAMP:LogDate} [%{WORD:LoggerName}] [%{NUMBER:ThreadNumber}] -%{LOGLEVEL:Level} - %{DATA:LogText}$"
]

		 if "specific" in [LogText] {
		    %{DATA:LogTag} 
		  }
        }

}

And got this:

:Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 7, column 8 (byte 1747) after filter{\n grok {\n match => [\n...."

[Badger]

Try this

filter {
    grok { match => [ "message", "%{DATESTAMP:LogDate} \[%{WORD:LoggerName}\] \[%{NUMBER:ThreadNumber}\] %{LOGLEVEL:Level} - %{DATA:LogText}$" ] }
    if "specific" in [LogText] {
        mutate { add_tag => [ "someTag" ] }
    }
}

[DanaDascalu]

Works perfectly, thanks!