Separating fields using grok with custom patterns

Hello! I have a problem while parsing my log file: I can not get the separated fields in stdout using ruby debug. The pattern is okay while using DevTools or the recommended debugger. also I have different log lines which have different patterns and I am using grok with an array of match. Here is a line from my log file:
[2020-10-12T21:04:15.090] [ERROR] Log - test-include : body: {"FirstName":"sdfsfd","MiddleName":"","LastName":"sfsf"} error: [object Object], status: 400

I would like to separate "body" fields in logstash output lile so:

{ 
  "first_name":""
  "last_name":""
  ...
}

or nested like this:

body {
     "first_name":""
     ...
}

here is my grok pattern:

\[%{TIMESTAMP_ISO8601:timestamp}\] \[%{LOGLEVEL:log_level}\] %{MSG:log_type} - %{MSG:action}-%{MSG:error_on} : %{MSG:info_type}: %{PERSON} error: %{MSG:error_message}, status: %{NUMBER:status_code:int}

and this is my custom grok pattern:

MSG %{DATA}
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?
    
PERSON {"FirstName":\"%{PERSONDATA:first_name}\","MiddleName":\"%{PERSONDATA:middle_name}\","LastName":\"%{PERSONDATA:last_name}\"}

PERSONDATA %{DATA}

here is the output by debugger:
{ "timestamp": [ [ "2020-10-12T21:04:15.090" ] ], "log_level": [ [ "ERROR" ] ], "log_type": [ [ "Log" ] ], "action": [ [ "test" ] ], "error_on": [ [ "include" ] ], "info_type": [ [ "body" ] ], "first_name": [ [ "sdfsfd" ] ], "middle_name": [ [ "" ] ], "last_name": [ [ "sfsf" ] ], "error_message": [ [ "[object Object]" ] ], "status_code": [ ] }

this is the logstash config:

input {
    file {
	path => "/path/to/log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}

filter {
    grok {
	patterns_dir => ["/path/to/patterns"]
        match => { "message" => ["pattern1","pattern2",...,"mentioned_pattern"]}
        remove_field => ["message","@timestamp"]
    }
}

output {
    stdout { codec => rubydebug }
}

here is the picture:


I would be thankful if anyone could help me with this.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.