# Create a conf file using grok or json filter
**URL:** https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903
**Category:** Logstash
**Created:** [November 18, 2020, 9:31pm UTC](https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903 "2020-11-18T21:31:39Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Omair](https://discuss.elastic.co/u/Omair)
#### Post date: [November 18, 2020, 9:31pm UTC](https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903/1 "2020-11-18T21:31:39Z")
Hi.. I'm new to elastic stack and trying to learn using filters to structure my data but unfortunately not getting much help.
Below is the application log
2020-11-19T00:08:34.627+0500 INFO 141799 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605726514602,"formattedTime":"2020-11-19T00:08:34.602+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"975d295ddd3228f50e7f4f2af4211a05","serviceName":"Test Logging","serviceUri":"/testlogging","totalFrontendLatency":23,"totalBackendLatency":23,"isPolicySuccessful":true,"isPolicyViolation":false,"isRoutingFailure":false}
I want to see this log in the below format
"time" : 1518734278251,
"formattedTime" : "2018-02-15T14:37:58.251-08:00",
"nodeId" : "48b5a4bc60fd4db191ddd5258eab0a35",
"nodeName" : "Gateway1",
"nodeIp" : "10.242.45.121",
"serviceId" : "13df5d4767ea633273940598aa2f323c",
"serviceName" : "Test Service 1",
"serviceUri" : "/test1",
"totalFrontendLatency" : 234,
"totalBackendLatency" : 123,
"isPolicySuccessful": true,
"isPolicyViolation" : false,
"isRoutingFailure" : false
I want someone to help me writing the logstash.conf file to achieve this.
Any help would be appreciated.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 18, 2020, 9:51pm UTC](https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903/2 "2020-11-18T21:51:19Z")
Can you share what you have so far?
Please also format your code/logs/config using the `` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂
---
### Author:  [@Omair](https://discuss.elastic.co/u/Omair)
#### Post date: [November 19, 2020, 11:05am UTC](https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903/3 "2020-11-19T11:05:17Z")
Actual Log Entry
` 2020-11-19T00:08:34.627+0500 INFO 141799 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605726514602,"formattedTime":"2020-11-19T00:08:34.602+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"975d295ddd3228f50e7f4f2af4211a05","serviceName":"Test Logging","serviceUri":"/testlogging","totalFrontendLatency":23,"totalBackendLatency":23,"isPolicySuccessful":true,"isPolicyViolation":false,"isRoutingFailure":false}`
The way i want it
> "time" : 1518734278251,
> "formattedTime" : "2018-02-15T14:37:58.251-08:00",
> "nodeId" : "48b5a4bc60fd4db191ddd5258eab0a35",
> "nodeName" : "Gateway1",
> "nodeIp" : "10.242.45.121",
> "serviceId" : "13df5d4767ea633273940598aa2f323c",
> "serviceName" : "Test Service 1",
> "serviceUri" : "/test1",
> "totalFrontendLatency" : 234,
> "totalBackendLatency" : 123,
> "isPolicySuccessful": true,
> "isPolicyViolation" : false,
> "isRoutingFailure" : false
My logstash.conf file
> input {
> beats {
> port =\> "5044"
> }
> }
>
> ```
> filter {
> json {
> source => "message"
> }
> }
>
> output {
> elasticsearch {
> hosts => "http://:9200"
> index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
> }
>
> stdout{}
> }
>
> ```
when im running with this conf file im getting \_jsonparsefailure tag in my logs.
---
### Author:  [@ylasri](https://discuss.elastic.co/u/ylasri)
#### Post date: [November 19, 2020, 11:17am UTC](https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903/4 "2020-11-19T11:17:29Z")
Your message is not fully jsn, you need to use grok filter first then you access to the json payload of your message
```auto
grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{NUMBER:process_id} %{JAVACLASS:logclass}: %{INT:int_value}: metrics: %{GREEDYDATA:json_payload}"}}
json { source => "json_payload" }
```
I would recommand to use Kibana Grok debugger to quickly test your pattern
---
### Author:  [@Omair](https://discuss.elastic.co/u/Omair)
#### Post date: [November 19, 2020, 1:57pm UTC](https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903/5 "2020-11-19T13:57:28Z")
Thanks for your help Yassine. Much appreciated!
There are some extra spaces in the log between INFO and process\_id by default.. When I used your pattern i was unable to get the response in the debugger but then i manually removed the spaces and then the pattern worked. How I can remove these extra spaces using a filter? Below is my updated conf file and the output im getting. still getting \_jsonparsefailure tag
```
> input {
> beats {
> port => "5044"
> }
> }
>
> filter {
> grok {
> match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{NUMBER:process_id} %{JAVACLASS:logclass}: %{INT:int_value}: metrics: %{GREEDYDATA:json_payload}" }
> }
> json {
> source => "json_payload"
> }
> }
>
> output {
> elasticsearch {
> hosts => "http://elasticsearch:9200"
> index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
> }
>
> stdout{}
> }
```
output im getting
> "message" : "2020-11-19T14:37:21.525+0500 INFO 146061 com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: metrics: {"time":1605778641452,"formattedTime":"2020-11-19T14:37:21.452+05:00","nodeId":"585e20c505994785aa4004971b36b94b","nodeName":"Gateway1","nodeIp":"192.168.1.171","serviceId":"ce9fee8e5a20855a272555ab9c37a964","serviceName":"assertionMessageCode\_7203","serviceUri":"/audit\_7203","totalFrontendLatency":70,"totalBackendLatency":69,"isPolicySuccessful":false,"isPolicyViolation":true,"isRoutingFailure":false}",
> "tags" : [
> "APIGW",
> "beats\_input\_codec\_plain\_applied",
> "\_jsonparsefailure"
> ],
---
### Author:  [@ylasri](https://discuss.elastic.co/u/ylasri)
#### Post date: [November 19, 2020, 2:09pm UTC](https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903/6 "2020-11-19T14:09:23Z")
\s\* is the pattern to much any number of spaces
```auto
%{TIMESTAMP_ISO8601:timestamp}\s*%{LOGLEVEL:log_level}\s*%{NUMBER:process_id}\s*%{JAVACLASS:logclass}:\s*%{INT:int_value}:\s*metrics:\s*%{GREEDYDATA:json_payload}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 17, 2020, 2:09pm UTC](https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903/7 "2020-12-17T14:09:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.