Parsing logs using Regex pattern in logstash filter

Elastic Stack Logstash
1

Hi,

I am using logstash to ingest data into the Elasticsearch using a txt file.
I build a regex query to extract msg-id field from the logs, but msg-id field is not getting extracted though the regex query is working fine.

Any changes/suggestions are required in the regex query for msg-id field.
The original msg-id field value is "MSGID: " which is not getting mapped in output.

regex testing link-https://regex101.com/r/R1ohvI/1

Original FIlter-
filter-

filter{
 grok {
        break_on_match => false
        match => {
            "message" => [
                "(MSGID|Message Id)\s*:\s*(?<msgid>(?:(?!Mobile)[A-Z0-9a-z+-]){7,36})",
                ]}}}

Output-

{
            "host" => "dev",
         "message" => "10/11/2021 5:49:43 PM,MSGID: <CSError>Mobile_Number: <>Job_no:<>User_id<> mstrGateWay <>\r"
}

After adding "<>"sign in msg id field, I am able to get msg id field in output, but I don't want output, instead I want this output "CSError"

Updated FIlter-

 "message" => [
                "(MSGID|Message Id)\s*:\s*(?<msgid>(?:(?!Mobile)[A-Z0-9a-z+-<>]){7,36})",

Output-

 "host" => "dev",
 "message" => "10/11/2021 5:49:43 PM,MSGID: <CSError>Mobile_Number: <>Job_no:<>User_id<> mstrGateWay <>\r",
"MSGID" => "<CSError>"

Thank you in Adance

2

Try this grok pattern instead:
%{DATE} %{TIME} (AM|PM),MSGID: <%{NOTSPACE:msgid}>Mobile_Number: <?%{NOTSPACE:mobile_number}>Job_no:<?%{NOTSPACE:user_id} mstrGateWay <?%{NOTSPACE:mst_gate_way}>\\r

It's not perfect, and needs a bit of tweaking but if you use https://grokdebug.herokuapp.com/
then it will be a lot easier!

3

Thanks @AquaX ,I will check on it,

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.