Logstash-filter-grok repeatedly

Elastic Stack Logstash
1

Good time of the day.
I'm parsing log files like these
2017.04.25 09:55:22 [LOG] [4cdb70e9-7e41-48c6-a3b4-889790630641] [10.4.26.21:19285] Open
2017.04.25 09:55:22 [LOG] [4cdb70e9-7e41-48c6-a3b4-889790630641] [10.4.26.21:19285] Recieve !SAPI!CSG|000000000000||
2017.04.25 09:55:22 [LOG] [4cdb70e9-7e41-48c6-a3b4-889790630641] [10.4.26.21:19285] Message: CSG|000000000000||
2017.04.25 09:55:22 [LOG] [4cdb70e9-7e41-48c6-a3b4-889790630641] [10.4.26.21:19285] SAPISubscriberGet: Start for 000000000000;
2017.04.25 09:55:22 [LOG] [4cdb70e9-7e41-48c6-a3b4-889790630641] [10.4.26.21:19285] SAPISubscriberGet:response:000000000000~28551349~0~30188079~6985430049~False~51005261~2~26.07.2012 9:22:59~False~2~2~~; Success.
2017.04.25 09:55:22 [LOG] [4cdb70e9-7e41-48c6-a3b4-889790630641] [10.4.26.21:19285] Query: 1
2017.04.25 09:55:22 [LOG] [4cdb70e9-7e41-48c6-a3b4-889790630641] [10.4.26.21:19285] Back answer: !SAPI!1|000000000000~28551349~0~30188079~6985430049~False~51005261~2~26.07.2012 9:22:59~False~2~2~~
2017.04.25 09:55:22 [LOG] [4cdb70e9-7e41-48c6-a3b4-889790630641] [10.4.26.21:19285] Close

Here is my conf file content
} else if [doc_type] == "sapi" {
grok {
match => { "message" => "(?[^[]+)(\s)+[%{WORD:loglevel}](\s)+[(?[\w\d-]+)](\s)+[%
{IP:ip}:%{NUMBER:port}](\s)+(?\w+(\s)?[\w]+)(:(\s)+%{NUMBER:command_code})?(\s+!SAPI!%
{WORD:cmd}(|(?<input_params_str>.+))?)?(:\s+Start for (?<identifier_str>.+))?(:\s!SAPI!(?<back_answer>.+))?(:(\s)?
(?<temp_message>[\w\d;:~\s-,.^|]+))?" }
}
date {
match => [ "logtime", "yyyy.MM.dd HH:mm:ss" ]
target => "timestamp"
}
if "_grokparsefailure" in [tags] {
drop { }
} else {
if [command] == "Receive"{
if [cmd] in ["CSG", "CSR", "CSUBRETFPHB", "CSCFOWB"] {
grok {
match => { "input_params_str" => "(?\d+)" }
}
} else if [cmd] in ["CACCRETR"] {
grok {
match => { "input_params_str" => "(?\d+)" }
}
} else if [cmd] in ["CSGETED", "CCCFBS", "CSSPO", "CSAOI"] {
grok {
match => { "input_params_str" => "(?\d+)" }
}
}
} else if [command] == "Message" {
if [msg_cmd] in ["CSG", "CSR", "CSUBRETFPHB", "CSCFOWB"] {
grok {
match => { "temp_message" => "(?\d+)" }
}
} else if [msg_cmd] in ["CACCRETR"] {
grok {
match => { "temp_message" => "(?\d+)" }
}
} else if [msg_cmd] in ["CSGETED", "CCCFBS", "CSSPO", "CSAOI"] {
grok {
match => { "temp_message" => "(?\d+)" }
}
}
}
mutate {
remove_field => ["message", "logtime", "beat", "tags", "source", "type"]
}
}

Everything goes fine untill I use grok again. What Am I doing wrong? Does grok support repetitive usage?
Thank you beforehand

2

What do you mean by using grok again? Where is it exactly?

And can you use code style format? It's really hard to correctly read it as it is right now

3

Nico-DF, I did as you requested.

Here is cut version

grok {
   match => { "message" => "(?<logtime>[^\[]+)(\s)+\[%{WORD:loglevel}\](\s)+\[(?<guid>[\w\d-]+)\](\s)+\[%{IP:ip}:%
{NUMBER:port}\](\s)+(?<command>\w+(\s)?[\w]+)(:(\s)+%{NUMBER:command_code})?(\s+!SAPI!%{WORD:cmd}(\|
(?<input_params_str>.+))?)?(:\s+Start for (?<identifier_str>.+))?(:\s!SAPI!(?<back_answer>.+))?(:(\s)?(?
<temp_message>[\w\d;:~\s\-,.\^|]+))?" }
}
if [command] == "Receive"{
    if [cmd] in ["CSG", "CSR", "CSUBRETFPHB", "CSCFOWB"] {
        grok {
            match => { "input_params_str" => "(?<msisdn>\d+)" }
        }
    }
}
4

What do you have in output? (using stdout { codec => rubydebug {} })
Does it tell you _grokparsefailed? or is there any other error?

By the way, test:

if [command] == "Recieve "

(typo)

5

It was caused by misspelling when writing the logs. After changing "Receive" to "Recieve", results are good )). Thank you for help

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.