Hello! I have a problem while parsing my log file: I can not get the separated fields in stdout using ruby debug. The pattern is okay while using DevTools or the recommended debugger. also I have different log lines which have different patterns and I am using grok with an array of match. Here is a line from my log file:
[2020-10-12T21:04:15.090] [ERROR] Log - test-include : body: {"FirstName":"sdfsfd","MiddleName":"","LastName":"sfsf"} error: [object Object], status: 400
I would like to separate "body" fields in logstash output lile so:
{
"first_name":""
"last_name":""
...
}
or nested like this:
body {
"first_name":""
...
}
here is my grok pattern:
\[%{TIMESTAMP_ISO8601:timestamp}\] \[%{LOGLEVEL:log_level}\] %{MSG:log_type} - %{MSG:action}-%{MSG:error_on} : %{MSG:info_type}: %{PERSON} error: %{MSG:error_message}, status: %{NUMBER:status_code:int}
and this is my custom grok pattern:
MSG %{DATA}
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?
PERSON {"FirstName":\"%{PERSONDATA:first_name}\","MiddleName":\"%{PERSONDATA:middle_name}\","LastName":\"%{PERSONDATA:last_name}\"}
PERSONDATA %{DATA}
here is the output by debugger:
{ "timestamp": [ [ "2020-10-12T21:04:15.090" ] ], "log_level": [ [ "ERROR" ] ], "log_type": [ [ "Log" ] ], "action": [ [ "test" ] ], "error_on": [ [ "include" ] ], "info_type": [ [ "body" ] ], "first_name": [ [ "sdfsfd" ] ], "middle_name": [ [ "" ] ], "last_name": [ [ "sfsf" ] ], "error_message": [ [ "[object Object]" ] ], "status_code": [ ] }
this is the logstash config:
input {
file {
path => "/path/to/log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
patterns_dir => ["/path/to/patterns"]
match => { "message" => ["pattern1","pattern2",...,"mentioned_pattern"]}
remove_field => ["message","@timestamp"]
}
}
output {
stdout { codec => rubydebug }
}
here is the picture:
I would be thankful if anyone could help me with this.