Create a conf file using grok or json filter

[Omair (Muhammad Omair Siddiqui)]

Hi.. I'm new to elastic stack and trying to learn using filters to structure my data but unfortunately not getting much help.

Below is the application log

2020-11-19T00:08:34.627+0500 INFO 141799 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605726514602,"formattedTime":"2020-11-19T00:08:34.602+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"975d295ddd3228f50e7f4f2af4211a05","serviceName":"Test Logging","serviceUri":"/testlogging","totalFrontendLatency":23,"totalBackendLatency":23,"isPolicySuccessful":true,"isPolicyViolation":false,"isRoutingFailure":false}

I want to see this log in the below format
"time" : 1518734278251,
"formattedTime" : "2018-02-15T14:37:58.251-08:00",
"nodeId" : "48b5a4bc60fd4db191ddd5258eab0a35",
"nodeName" : "Gateway1",
"nodeIp" : "10.242.45.121",
"serviceId" : "13df5d4767ea633273940598aa2f323c",
"serviceName" : "Test Service 1",
"serviceUri" : "/test1",
"totalFrontendLatency" : 234,
"totalBackendLatency" : 123,
"isPolicySuccessful": true,
"isPolicyViolation" : false,
"isRoutingFailure" : false

I want someone to help me writing the logstash.conf file to achieve this.
Any help would be appreciated.

[warkolm (Mark Walkom)]

Can you share what you have so far?

Please also format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you

[Omair (Muhammad Omair Siddiqui)]

Actual Log Entry

2020-11-19T00:08:34.627+0500 INFO 141799 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605726514602,"formattedTime":"2020-11-19T00:08:34.602+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"975d295ddd3228f50e7f4f2af4211a05","serviceName":"Test Logging","serviceUri":"/testlogging","totalFrontendLatency":23,"totalBackendLatency":23,"isPolicySuccessful":true,"isPolicyViolation":false,"isRoutingFailure":false}

The way i want it

"time" : 1518734278251,
"formattedTime" : "2018-02-15T14:37:58.251-08:00",
"nodeId" : "48b5a4bc60fd4db191ddd5258eab0a35",
"nodeName" : "Gateway1",
"nodeIp" : "10.242.45.121",
"serviceId" : "13df5d4767ea633273940598aa2f323c",
"serviceName" : "Test Service 1",
"serviceUri" : "/test1",
"totalFrontendLatency" : 234,
"totalBackendLatency" : 123,
"isPolicySuccessful": true,
"isPolicyViolation" : false,
"isRoutingFailure" : false

My logstash.conf file

input {
beats {
port => "5044"
}
}

filter {
        json {
                source => "message"
        }
}

output {
        elasticsearch {
                hosts => "http://<elaseticsearch-IP>:9200"
                index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        }

        stdout{}
}

when im running with this conf file im getting _jsonparsefailure tag in my logs.

[ylasri (Yassine LASRI)]

Your message is not fully jsn, you need to use grok filter first then you access to the json payload of your message

grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{NUMBER:process_id} %{JAVACLASS:logclass}: %{INT:int_value}: metrics: %{GREEDYDATA:json_payload}"}}
json { source => "json_payload" }

I would recommand to use Kibana Grok debugger to quickly test your pattern

[Omair (Muhammad Omair Siddiqui)]

Thanks for your help Yassine. Much appreciated!
There are some extra spaces in the log between INFO and process_id by default.. When I used your pattern i was unable to get the response in the debugger but then i manually removed the spaces and then the pattern worked. How I can remove these extra spaces using a filter? Below is my updated conf file and the output im getting. still getting _jsonparsefailure tag

> input {
>         beats {
>                 port => "5044"
>         }
> }
> 
> filter {
>         grok {
>                 match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{NUMBER:process_id} %{JAVACLASS:logclass}: %{INT:int_value}: metrics: %{GREEDYDATA:json_payload}" }
>         }
>         json {
>                 source => "json_payload"
>         }
> }
> 
> output {
>         elasticsearch {
>                 hosts => "http://elasticsearch:9200"
>                 index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
>         }
> 
>         stdout{}
> }

output im getting

"message" : "2020-11-19T14:37:21.525+0500 INFO 146061 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605778641452,"formattedTime":"2020-11-19T14:37:21.452+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"ce9fee8e5a20855a272555ab9c37a964","serviceName":"assertionMessageCode_7203","serviceUri":"/audit_7203","totalFrontendLatency":70,"totalBackendLatency":69,"isPolicySuccessful":false,"isPolicyViolation":true,"isRoutingFailure":false}",
"tags" : [
"APIGW",
"beats_input_codec_plain_applied",
"_jsonparsefailure"
],

[ylasri (Yassine LASRI)]

\s* is the pattern to much any number of spaces

%{TIMESTAMP_ISO8601:timestamp}\s*%{LOGLEVEL:log_level}\s*%{NUMBER:process_id}\s*%{JAVACLASS:logclass}:\s*%{INT:int_value}:\s*metrics:\s*%{GREEDYDATA:json_payload}