Create a conf file using grok or json filter

Hi.. I'm new to elastic stack and trying to learn using filters to structure my data but unfortunately not getting much help.

Below is the application log

2020-11-19T00:08:34.627+0500 INFO 141799 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605726514602,"formattedTime":"2020-11-19T00:08:34.602+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"975d295ddd3228f50e7f4f2af4211a05","serviceName":"Test Logging","serviceUri":"/testlogging","totalFrontendLatency":23,"totalBackendLatency":23,"isPolicySuccessful":true,"isPolicyViolation":false,"isRoutingFailure":false}

I want to see this log in the below format
"time" : 1518734278251,
"formattedTime" : "2018-02-15T14:37:58.251-08:00",
"nodeId" : "48b5a4bc60fd4db191ddd5258eab0a35",
"nodeName" : "Gateway1",
"nodeIp" : "10.242.45.121",
"serviceId" : "13df5d4767ea633273940598aa2f323c",
"serviceName" : "Test Service 1",
"serviceUri" : "/test1",
"totalFrontendLatency" : 234,
"totalBackendLatency" : 123,
"isPolicySuccessful": true,
"isPolicyViolation" : false,
"isRoutingFailure" : false

I want someone to help me writing the logstash.conf file to achieve this.
Any help would be appreciated.

Can you share what you have so far?

Please also format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

Actual Log Entry

2020-11-19T00:08:34.627+0500 INFO 141799 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605726514602,"formattedTime":"2020-11-19T00:08:34.602+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"975d295ddd3228f50e7f4f2af4211a05","serviceName":"Test Logging","serviceUri":"/testlogging","totalFrontendLatency":23,"totalBackendLatency":23,"isPolicySuccessful":true,"isPolicyViolation":false,"isRoutingFailure":false}

The way i want it

"time" : 1518734278251,
"formattedTime" : "2018-02-15T14:37:58.251-08:00",
"nodeId" : "48b5a4bc60fd4db191ddd5258eab0a35",
"nodeName" : "Gateway1",
"nodeIp" : "10.242.45.121",
"serviceId" : "13df5d4767ea633273940598aa2f323c",
"serviceName" : "Test Service 1",
"serviceUri" : "/test1",
"totalFrontendLatency" : 234,
"totalBackendLatency" : 123,
"isPolicySuccessful": true,
"isPolicyViolation" : false,
"isRoutingFailure" : false

My logstash.conf file

input {
beats {
port => "5044"
}
}

filter {
        json {
                source => "message"
        }
}

output {
        elasticsearch {
                hosts => "http://<elaseticsearch-IP>:9200"
                index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        }

        stdout{}
}

when im running with this conf file im getting _jsonparsefailure tag in my logs.

Your message is not fully jsn, you need to use grok filter first then you access to the json payload of your message

grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{NUMBER:process_id} %{JAVACLASS:logclass}: %{INT:int_value}: metrics: %{GREEDYDATA:json_payload}"}}
json { source => "json_payload" }

I would recommand to use Kibana Grok debugger to quickly test your pattern

Thanks for your help Yassine. Much appreciated!
There are some extra spaces in the log between INFO and process_id by default.. When I used your pattern i was unable to get the response in the debugger but then i manually removed the spaces and then the pattern worked. How I can remove these extra spaces using a filter? Below is my updated conf file and the output im getting. still getting _jsonparsefailure tag

> input {
>         beats {
>                 port => "5044"
>         }
> }
> 
> filter {
>         grok {
>                 match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{NUMBER:process_id} %{JAVACLASS:logclass}: %{INT:int_value}: metrics: %{GREEDYDATA:json_payload}" }
>         }
>         json {
>                 source => "json_payload"
>         }
> }
> 
> output {
>         elasticsearch {
>                 hosts => "http://elasticsearch:9200"
>                 index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
>         }
> 
>         stdout{}
> }

output im getting

"message" : "2020-11-19T14:37:21.525+0500 INFO 146061 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605778641452,"formattedTime":"2020-11-19T14:37:21.452+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"ce9fee8e5a20855a272555ab9c37a964","serviceName":"assertionMessageCode_7203","serviceUri":"/audit_7203","totalFrontendLatency":70,"totalBackendLatency":69,"isPolicySuccessful":false,"isPolicyViolation":true,"isRoutingFailure":false}",
"tags" : [
"APIGW",
"beats_input_codec_plain_applied",
"_jsonparsefailure"
],

\s* is the pattern to much any number of spaces

%{TIMESTAMP_ISO8601:timestamp}\s*%{LOGLEVEL:log_level}\s*%{NUMBER:process_id}\s*%{JAVACLASS:logclass}:\s*%{INT:int_value}:\s*metrics:\s*%{GREEDYDATA:json_payload}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.