Logstash - parse/ filter lines having a specific string

Elastic Stack Logstash
1

Hi All,

We are running ELK 7.6.2 stack in our environment.

The log generated by our application throws out messages with different Log Levels. Please see below example:

2021-10-31 19:00:01,062|DEBUG|DispatcherServlet|anonymous-1.2.3.4.106-TVZ-PCVLSB69GCBUWD1NCMAVMYJJJ4YC6KG1744901501|Exiting from "ASYNC" dispatch
2021-10-31 19:00:01,059| INFO|HttpRequestUtil|anonymous-1.2.3.4.106-TVZ-PCVLSB69GCBUWD1NCMAVMYJJJ4YC6KG1744901501|ERROR STATUS CODE 500
2021-10-31 19:00:01,060| INFO|ErrorController|anonymous-1.2.3.4.106-TVZ-PCVLSB69GCBUWD1NCMAVMYJJJ4YC6KG1744901501|ERROR ASYNC HANDLER async isCommitted=false
2021-10-31 19:20:01,286| INFO|HttpRequestUtil|anonymous-1.2.3.4-1-01-CV-PCVY73MMUJJ3LBIMAMGJ97EBIPLWJWG1745536201@1-1163547#17|ERROR STATUS CODE 401
2021-10-31 19:20:32,606|ERROR|SlUtil|1.2.3.4-TVZ-PCVZXJECFBRFBM43FFEREJUAFEUWUMG4732288802@1-1163852#17|Parsing the request failed

Filebeat agent running on the server pushes this content to Logstash.

I want Logstash to just parse/ filter the line that has "ERROR" as Log Level (Last line in the sample log above) and push it to Elasticsearch. The rest should be discarded.

Please guide on how this could be achieved.

Thanks

2

It could be as simple as

if "|ERROR|" not in [message] { drop {} }
3

Thanks @Badger as always!

I introduced this in my logtstash config file as the following snippet:

        if [type] == "tv_proxy_log"  {
                mutate {
                        split => ["message", "|"]
                        add_field =>{
                           "createdTime" => "%{[message][0]}"
                           "logLevel" => "%{[message][1]}"
                           "className" => "%{[message][2]}"
                           "reqID" => "%{[message][3]}"
                           "message" => "%{[message][4]}"

                       }
                }
        if "|ERROR|" not in [message] { drop {} }

                      -------------------------------
                      -------------------------------
                      -------------------------------

As a result it has stopped showing "all" messages in Kibana, whereas I expect it to show the following message:

2021-11-03 16:34:49,208|ERROR|UserController|TV-UCVZ3CJK4CCJAJ9JCT4DFYFCBZM6FWG3013461403|Not sufficient auth level to get user profile

Please guide

4

After this [message] is an array that does not contain |, so everything gets dropped. Try

if [logLevel] != "ERROR" { drop {} }
5

Thanks @Badger. It worked!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.