# JSON Parse an already parsed field **URL:** \ **Category:** Logstash\ **Created:** [September 3, 2018, 3:54am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006 "2018-09-03T03:54:25Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![jibsonline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jibsonline/32/63154_2.png) [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Post date:** [September 3, 2018, 3:54am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/1 "2018-09-03T03:54:26Z")
I have the following field parsed using JSON filter ``` json { source => "log" target => "request" skip_on_invalid_json => true } ``` But there is a field in "request" which still prints as JSON String `request.headers.x-user-agent : {"platform":"android","app_version":"2.5.0109","device_id":"24baa5b0bee94d8f","model":"OnePlus ONEPLUS A3003","os_version":26,"user_id":"235978867"}` I am trying to split the field again with below filter ``` json { source => "[request][headers][x-user-agent]" skip_on_invalid_json => true } ``` But it is not working as expected. Is there any limitation in parsing an already parsed field? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 3, 2018, 6:09am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/2 "2018-09-03T06:09:36Z")
Please show example what the event looks like. Use a `stdout { codec => rubydebug }` output to dump the raw event (or copy/paste the JSON from Kibana's JSON tab). ---
**Author:** ![jibsonline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jibsonline/32/63154_2.png) [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Post date:** [September 3, 2018, 8:21am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/3 "2018-09-03T08:21:02Z")
Sample output: `[http-nio-8080-exec-3] INFO http.wire-log.writeRequest 79 - [6851535958919] [5209babb-9658-45a2-a652-7fee91b83ed9] {"origin":"remote","type":"request","correlation":"ac44a6921e00b575","protocol":"HTTP/1.1","remote":"172.18.3.59","method":"GET","uri":"http://bdmutualfund.xxxxxxxxx.com/mf/v1/mf-scheme/transaction-meta/INF247L01502","headers":{"accept":["application/json"],"accept-encoding":["gzip,deflate"],"app_version":["2.5.0109"],"authorization":["XXX"],"host":["bdmutualfund.xxxxxxxxx.com"],"user-agent":["Apache-HttpClient/4.5.4 (Java/1.8.0_171)"],"user_id":["2851110"],"x-amzn-trace-id":["Root=1-5b8cdf88-0125a7d07b389770abf34950"],"x-device-id":["d0450727884ae463"],"x-device-model":["Xiaomi Redmi Note 5 pro"],"x-forwarded-for":["172.18.3.83"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-os-version":["27"],"x-platform":["android"],"x-pmmodule-name":["mftransaction"],"x-request-id":["6851535958919"],"x-sso-token":["c6c94d9c-b4d5-477d-811e-377680247200"],"x-user-agent":["{\"platform\":\"android\",\"app_version\":\"2.5.0109\",\"device_id\":\"d0450727884ae463\",\"model\":\"Xiaomi Redmi Note 5 pro\",\"os_version\":27,\"user_id\":\"2851110\"}"]}}` My conf file is too large. I will add the important parts ``` grok { match => { "message" => "\[%{DATA:thread}\]\s+%{LOGLEVEL:severity}\s+%{DATA:class}\s+%{DATA:linenum}\s+-\s%{GREEDYDATA:log}" } } } if [message] =~ /http.wire-log.writeRequest/ { mutate { add_field => { "http_type" => "request" } } json { source => "log" target => "request" skip_on_invalid_json => true } json { source => "[request][headers][x-user-agent]" skip_on_invalid_json => true } json_encode { source => "[request][body]" } } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 3, 2018, 8:36am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/4 "2018-09-03T08:36:27Z")
I asked for the event that goes _out_ of Logstash, not what goes _in_. Use a `stdout { codec => rubydebug }` output to dump the raw event (or copy/paste the JSON from Kibana's JSON tab). ---
**Author:** ![jibsonline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jibsonline/32/63154_2.png) [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Post date:** [September 3, 2018, 8:50am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/5 "2018-09-03T08:50:10Z")
Sorry... Missed the JSON Tab part ``` { "_index": "java-prod-2018.09.03", "_type": "doc", "_id": "LcKbnmUBVAVv5iCHqjAR", "_version": 1, "_score": null, "_source": { "@timestamp": "2018-09-03T08:45:25.296Z", "linenum": "79", "source": "/home/springboot/logs/app.log", "thread": "http-nio-8080-exec-9", "class": "http.wire-log.writeRequest", "xresponseid": "a793c128-97f0-4a91-b642-b9b382c4c753", "http_type": "request", "xrequestid": "10171535964321", "@version": "1", "app": "mf-transaction", "severity": "INFO", "offset": 29915969, "beat": { "name": "ip-172-18-3-83", "hostname": "ip-172-18-3-83", "version": "6.2.3" }, "log": "{\"origin\":\"remote\",\"type\":\"request\",\"correlation\":\"c5aa4d8700683942\",\"protocol\":\"HTTP/1.1\",\"remote\":\"172.18.4.36\",\"method\":\"GET\",\"uri\":\"http://internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080/mftransaction/v1/246218764/history\",\"headers\":{\"accept-encoding\":[\"gzip\"],\"akamai-origin-hop\":[\"2\"],\"authorization\":[\"XXX\"],\"cache-control\":[\"no-cache, max-age=0\"],\"host\":[\"internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080\"],\"pragma\":[\"no-cache\"],\"true-client-ip\":[\"2405:204:1095:e1a0:f095:c663:af31:2347\"],\"user-agent\":[\"okhttp/3.10.0\"],\"via\":[\"1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)\"],\"x-akamai-config-log-detail\":[\"true\"],\"x-amzn-trace-id\":[\"Self=1-5b8cf4a2-53c4c1d6b5f017d63e31fe56;Root=1-5b8cf4a2-b35c43a2f0c3530b00332525\"],\"x-client-utc-offset\":[\"330\"],\"x-forwarded-for\":[\"2405:204:1095:e1a0:f095:c663:af31:2347, 49.44.130.71, 104.124.54.69, 104.124.54.69, 172.18.3.98\"],\"x-forwarded-port\":[\"8080\"],\"x-forwarded-proto\":[\"http\"],\"x-pmngx-key\":[\"904b36db3f10315953c3de1b918c0084\",\"904b36db3f10315953c3de1b918c0084\"],\"x-request-id\":[\"10171535964321\"],\"x-sso-token\":[\"10ed9c50-c9e4-4bec-9a0f-1f3b9eb37100\"],\"x-user-agent\":[\"{\\\"platform\\\":\\\"android\\\",\\\"app_version\\\":\\\"1.10.1088\\\",\\\"device_id\\\":\\\"c83396cbfbc06cc0\\\",\\\"model\\\":\\\"Xiaomi Redmi Note 5\\\",\\\"os_version\\\":25,\\\"ip_address\\\":\\\"0.0.0.0\\\",\\\"user_id\\\":\\\"246218764\\\"}\"]}}", "request": { "method": "GET", "remote": "172.18.4.36", "body": "null", "headers": { "x-akamai-config-log-detail": [ "true" ], "x-client-utc-offset": [ "330" ], "authorization": [ "XXX" ], "user-agent": [ "okhttp/3.10.0" ], "x-forwarded-proto": [ "http" ], "x-pmngx-key": [ "904b36db3f10315953c3de1b918c0084", "904b36db3f10315953c3de1b918c0084" ], "host": [ "internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080" ], "akamai-origin-hop": [ "2" ], "x-amzn-trace-id": [ "Self=1-5b8cf4a2-53c4c1d6b5f017d63e31fe56;Root=1-5b8cf4a2-b35c43a2f0c3530b00332525" ], "x-request-id": [ "10171535964321" ], "accept-encoding": [ "gzip" ], "x-forwarded-port": [ "8080" ], "via": [ "1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)" ], "x-sso-token": [ "10ed9c50-c9e4-4bec-9a0f-1f3b9eb37100" ], "x-user-agent": [ "{\"platform\":\"android\",\"app_version\":\"1.10.1088\",\"device_id\":\"c83396cbfbc06cc0\",\"model\":\"Xiaomi Redmi Note 5\",\"os_version\":25,\"ip_address\":\"0.0.0.0\",\"user_id\":\"246218764\"}" ], "cache-control": [ "no-cache, max-age=0" ], "pragma": [ "no-cache" ], "true-client-ip": [ "2405:204:1095:e1a0:f095:c663:af31:2347" ], "x-forwarded-for": [ "2405:204:1095:e1a0:f095:c663:af31:2347, 49.44.130.71, 104.124.54.69, 104.124.54.69, 172.18.3.98" ] }, "uri": "http://internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080/mftransaction/v1/246218764/history", "protocol": "HTTP/1.1", "origin": "remote", "type": "request", "correlation": "c5aa4d8700683942" }, "prospector": { "type": "log" }, "message": "[http-nio-8080-exec-9] INFO http.wire-log.writeRequest 79 - [10171535964321] [a793c128-97f0-4a91-b642-b9b382c4c753] {\"origin\":\"remote\",\"type\":\"request\",\"correlation\":\"c5aa4d8700683942\",\"protocol\":\"HTTP/1.1\",\"remote\":\"172.18.4.36\",\"method\":\"GET\",\"uri\":\"http://internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080/mftransaction/v1/246218764/history\",\"headers\":{\"accept-encoding\":[\"gzip\"],\"akamai-origin-hop\":[\"2\"],\"authorization\":[\"XXX\"],\"cache-control\":[\"no-cache, max-age=0\"],\"host\":[\"internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080\"],\"pragma\":[\"no-cache\"],\"true-client-ip\":[\"2405:204:1095:e1a0:f095:c663:af31:2347\"],\"user-agent\":[\"okhttp/3.10.0\"],\"via\":[\"1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)\"],\"x-akamai-config-log-detail\":[\"true\"],\"x-amzn-trace-id\":[\"Self=1-5b8cf4a2-53c4c1d6b5f017d63e31fe56;Root=1-5b8cf4a2-b35c43a2f0c3530b00332525\"],\"x-client-utc-offset\":[\"330\"],\"x-forwarded-for\":[\"2405:204:1095:e1a0:f095:c663:af31:2347, 49.44.130.71, 104.124.54.69, 104.124.54.69, 172.18.3.98\"],\"x-forwarded-port\":[\"8080\"],\"x-forwarded-proto\":[\"http\"],\"x-pmngx-key\":[\"904b36db3f10315953c3de1b918c0084\",\"904b36db3f10315953c3de1b918c0084\"],\"x-request-id\":[\"10171535964321\"],\"x-sso-token\":[\"10ed9c50-c9e4-4bec-9a0f-1f3b9eb37100\"],\"x-user-agent\":[\"{\\\"platform\\\":\\\"android\\\",\\\"app_version\\\":\\\"1.10.1088\\\",\\\"device_id\\\":\\\"c83396cbfbc06cc0\\\",\\\"model\\\":\\\"Xiaomi Redmi Note 5\\\",\\\"os_version\\\":25,\\\"ip_address\\\":\\\"0.0.0.0\\\",\\\"user_id\\\":\\\"246218764\\\"}\"]}}" }, "fields": { "@timestamp": [ "2018-09-03T08:45:25.296Z" ] }, "sort": [ 1535964325296 ] } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 3, 2018, 11:03am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/6 "2018-09-03T11:03:41Z")
`[request][headers][x-user-agent]` is an array field so you need to tell Logstash to parse its first element (i.e. `[request][headers][x-user-agent][0]`). ---
**Author:** ![jibsonline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jibsonline/32/63154_2.png) [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Post date:** [September 3, 2018, 11:26am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/7 "2018-09-03T11:26:53Z")
That did it!!!!!!! Thank you so much ---
**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\ **Post date:** [September 3, 2018, 11:33am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/8 "2018-09-03T11:33:31Z")
Hi Magnus, Any idea how to do this if you have multiple values in the `[x-user-agent]` field and the length is not quite known? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 3, 2018, 11:47am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/9 "2018-09-03T11:47:26Z")
> Any idea how to do this if you have multiple values in the `[x-user-agent]` field and the length is not quite known? You'd have to use a ruby filter. ---
**Author:** ![NerdSec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nerdsec/32/22056_2.png) [@NerdSec](https://discuss.elastic.co/u/NerdSec)\ **Post date:** [September 4, 2018, 5:33am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/10 "2018-09-04T05:33:59Z")
Hi magnus, I was able to parse the json object as seperate fields using the ruby filter, but i am not sure how to join the fields. Could you provide some pointers? Not sure if I should post it here or not. I have posted it in the original thread that I had created. > [@JSON array parsing in Logstash using the ruby filter](https://discuss.elastic.co/t/json-array-parsing-in-logstash-using-the-ruby-filter/146829/5): > > This is getting parsed properly according to Logstash, but it is getting indexed as a nested object in Elasticsearch. As they are not well supported in Kibana, I hope to create a flat structured array using the ruby filter. Here is my config: input { file { path =\> "/home/fire.log" sincedb\_path =\> "/dev/null" start\_position =\> "beginning" codec =\> json } } filter { ruby { code =\> " message\_array = event.get('[alert][explanation][cnc-services][cnc-service]') … ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 4, 2018, 6:11am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/11 "2018-09-04T06:11:05Z")
> Could you provide some pointers? Not in this thread. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 2, 2018, 6:11am UTC](https://discuss.elastic.co/t/json-parse-an-already-parsed-field/147006/12 "2018-10-02T06:11:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.