JSON Parse an already parsed field

Elastic Stack Logstash
1

I have the following field parsed using JSON filter

json {
        source => "log"
        target => "request"
        skip_on_invalid_json => true
      }

But there is a field in "request" which still prints as JSON String

request.headers.x-user-agent : {"platform":"android","app_version":"2.5.0109","device_id":"24baa5b0bee94d8f","model":"OnePlus ONEPLUS A3003","os_version":26,"user_id":"235978867"}

I am trying to split the field again with below filter

json {
        source => "[request][headers][x-user-agent]"
        skip_on_invalid_json => true
      }

But it is not working as expected. Is there any limitation in parsing an already parsed field?

2

Please show example what the event looks like. Use a stdout { codec => rubydebug } output to dump the raw event (or copy/paste the JSON from Kibana's JSON tab).

3

Sample output:

[http-nio-8080-exec-3] INFO http.wire-log.writeRequest 79 - [6851535958919] [5209babb-9658-45a2-a652-7fee91b83ed9] {"origin":"remote","type":"request","correlation":"ac44a6921e00b575","protocol":"HTTP/1.1","remote":"172.18.3.59","method":"GET","uri":"http://bdmutualfund.xxxxxxxxx.com/mf/v1/mf-scheme/transaction-meta/INF247L01502","headers":{"accept":["application/json"],"accept-encoding":["gzip,deflate"],"app_version":["2.5.0109"],"authorization":["XXX"],"host":["bdmutualfund.xxxxxxxxx.com"],"user-agent":["Apache-HttpClient/4.5.4 (Java/1.8.0_171)"],"user_id":["2851110"],"x-amzn-trace-id":["Root=1-5b8cdf88-0125a7d07b389770abf34950"],"x-device-id":["d0450727884ae463"],"x-device-model":["Xiaomi Redmi Note 5 pro"],"x-forwarded-for":["172.18.3.83"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-os-version":["27"],"x-platform":["android"],"x-pmmodule-name":["mftransaction"],"x-request-id":["6851535958919"],"x-sso-token":["c6c94d9c-b4d5-477d-811e-377680247200"],"x-user-agent":["{\"platform\":\"android\",\"app_version\":\"2.5.0109\",\"device_id\":\"d0450727884ae463\",\"model\":\"Xiaomi Redmi Note 5 pro\",\"os_version\":27,\"user_id\":\"2851110\"}"]}}

My conf file is too large. I will add the important parts

grok {
                match => { "message" => "\[%{DATA:thread}\]\s+%{LOGLEVEL:severity}\s+%{DATA:class}\s+%{DATA:linenum}\s+-\s%{GREEDYDATA:log}" }
       }
  }

if [message] =~ /http.wire-log.writeRequest/  {
  mutate {
    add_field => { "http_type" => "request" }
  }
   json {
        source => "log"
        target => "request"
        skip_on_invalid_json => true
      }
   json {
        source => "[request][headers][x-user-agent]"
        skip_on_invalid_json => true
      }
   json_encode { source => "[request][body]" }
  }
}
4

I asked for the event that goes out of Logstash, not what goes in. Use a stdout { codec => rubydebug } output to dump the raw event (or copy/paste the JSON from Kibana's JSON tab).

5

Sorry... Missed the JSON Tab part

{
  "_index": "java-prod-2018.09.03",
  "_type": "doc",
  "_id": "LcKbnmUBVAVv5iCHqjAR",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-09-03T08:45:25.296Z",
    "linenum": "79",
    "source": "/home/springboot/logs/app.log",
    "thread": "http-nio-8080-exec-9",
    "class": "http.wire-log.writeRequest",
    "xresponseid": "a793c128-97f0-4a91-b642-b9b382c4c753",
    "http_type": "request",
    "xrequestid": "10171535964321",
    "@version": "1",
    "app": "mf-transaction",
    "severity": "INFO",
    "offset": 29915969,
    "beat": {
      "name": "ip-172-18-3-83",
      "hostname": "ip-172-18-3-83",
      "version": "6.2.3"
    },
    "log": "{\"origin\":\"remote\",\"type\":\"request\",\"correlation\":\"c5aa4d8700683942\",\"protocol\":\"HTTP/1.1\",\"remote\":\"172.18.4.36\",\"method\":\"GET\",\"uri\":\"http://internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080/mftransaction/v1/246218764/history\",\"headers\":{\"accept-encoding\":[\"gzip\"],\"akamai-origin-hop\":[\"2\"],\"authorization\":[\"XXX\"],\"cache-control\":[\"no-cache, max-age=0\"],\"host\":[\"internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080\"],\"pragma\":[\"no-cache\"],\"true-client-ip\":[\"2405:204:1095:e1a0:f095:c663:af31:2347\"],\"user-agent\":[\"okhttp/3.10.0\"],\"via\":[\"1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)\"],\"x-akamai-config-log-detail\":[\"true\"],\"x-amzn-trace-id\":[\"Self=1-5b8cf4a2-53c4c1d6b5f017d63e31fe56;Root=1-5b8cf4a2-b35c43a2f0c3530b00332525\"],\"x-client-utc-offset\":[\"330\"],\"x-forwarded-for\":[\"2405:204:1095:e1a0:f095:c663:af31:2347, 49.44.130.71, 104.124.54.69, 104.124.54.69, 172.18.3.98\"],\"x-forwarded-port\":[\"8080\"],\"x-forwarded-proto\":[\"http\"],\"x-pmngx-key\":[\"904b36db3f10315953c3de1b918c0084\",\"904b36db3f10315953c3de1b918c0084\"],\"x-request-id\":[\"10171535964321\"],\"x-sso-token\":[\"10ed9c50-c9e4-4bec-9a0f-1f3b9eb37100\"],\"x-user-agent\":[\"{\\\"platform\\\":\\\"android\\\",\\\"app_version\\\":\\\"1.10.1088\\\",\\\"device_id\\\":\\\"c83396cbfbc06cc0\\\",\\\"model\\\":\\\"Xiaomi  Redmi Note 5\\\",\\\"os_version\\\":25,\\\"ip_address\\\":\\\"0.0.0.0\\\",\\\"user_id\\\":\\\"246218764\\\"}\"]}}",
    "request": {
      "method": "GET",
      "remote": "172.18.4.36",
      "body": "null",
      "headers": {
        "x-akamai-config-log-detail": [
          "true"
        ],
        "x-client-utc-offset": [
          "330"
        ],
        "authorization": [
          "XXX"
        ],
        "user-agent": [
          "okhttp/3.10.0"
        ],
        "x-forwarded-proto": [
          "http"
        ],
        "x-pmngx-key": [
          "904b36db3f10315953c3de1b918c0084",
          "904b36db3f10315953c3de1b918c0084"
        ],
        "host": [
          "internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080"
        ],
        "akamai-origin-hop": [
          "2"
        ],
        "x-amzn-trace-id": [
          "Self=1-5b8cf4a2-53c4c1d6b5f017d63e31fe56;Root=1-5b8cf4a2-b35c43a2f0c3530b00332525"
        ],
        "x-request-id": [
          "10171535964321"
        ],
        "accept-encoding": [
          "gzip"
        ],
        "x-forwarded-port": [
          "8080"
        ],
        "via": [
          "1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)"
        ],
        "x-sso-token": [
          "10ed9c50-c9e4-4bec-9a0f-1f3b9eb37100"
        ],
        "x-user-agent": [
          "{\"platform\":\"android\",\"app_version\":\"1.10.1088\",\"device_id\":\"c83396cbfbc06cc0\",\"model\":\"Xiaomi  Redmi Note 5\",\"os_version\":25,\"ip_address\":\"0.0.0.0\",\"user_id\":\"246218764\"}"
        ],
        "cache-control": [
          "no-cache, max-age=0"
        ],
        "pragma": [
          "no-cache"
        ],
        "true-client-ip": [
          "2405:204:1095:e1a0:f095:c663:af31:2347"
        ],
        "x-forwarded-for": [
          "2405:204:1095:e1a0:f095:c663:af31:2347, 49.44.130.71, 104.124.54.69, 104.124.54.69, 172.18.3.98"
        ]
      },
      "uri": "http://internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080/mftransaction/v1/246218764/history",
      "protocol": "HTTP/1.1",
      "origin": "remote",
      "type": "request",
      "correlation": "c5aa4d8700683942"
    },
    "prospector": {
      "type": "log"
    },
    "message": "[http-nio-8080-exec-9] INFO  http.wire-log.writeRequest 79 - [10171535964321] [a793c128-97f0-4a91-b642-b9b382c4c753] {\"origin\":\"remote\",\"type\":\"request\",\"correlation\":\"c5aa4d8700683942\",\"protocol\":\"HTTP/1.1\",\"remote\":\"172.18.4.36\",\"method\":\"GET\",\"uri\":\"http://internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080/mftransaction/v1/246218764/history\",\"headers\":{\"accept-encoding\":[\"gzip\"],\"akamai-origin-hop\":[\"2\"],\"authorization\":[\"XXX\"],\"cache-control\":[\"no-cache, max-age=0\"],\"host\":[\"internal-transaction-backend-prod-in-1242711116.ap-south-1.elb.amazonaws.com:8080\"],\"pragma\":[\"no-cache\"],\"true-client-ip\":[\"2405:204:1095:e1a0:f095:c663:af31:2347\"],\"user-agent\":[\"okhttp/3.10.0\"],\"via\":[\"1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)\"],\"x-akamai-config-log-detail\":[\"true\"],\"x-amzn-trace-id\":[\"Self=1-5b8cf4a2-53c4c1d6b5f017d63e31fe56;Root=1-5b8cf4a2-b35c43a2f0c3530b00332525\"],\"x-client-utc-offset\":[\"330\"],\"x-forwarded-for\":[\"2405:204:1095:e1a0:f095:c663:af31:2347, 49.44.130.71, 104.124.54.69, 104.124.54.69, 172.18.3.98\"],\"x-forwarded-port\":[\"8080\"],\"x-forwarded-proto\":[\"http\"],\"x-pmngx-key\":[\"904b36db3f10315953c3de1b918c0084\",\"904b36db3f10315953c3de1b918c0084\"],\"x-request-id\":[\"10171535964321\"],\"x-sso-token\":[\"10ed9c50-c9e4-4bec-9a0f-1f3b9eb37100\"],\"x-user-agent\":[\"{\\\"platform\\\":\\\"android\\\",\\\"app_version\\\":\\\"1.10.1088\\\",\\\"device_id\\\":\\\"c83396cbfbc06cc0\\\",\\\"model\\\":\\\"Xiaomi  Redmi Note 5\\\",\\\"os_version\\\":25,\\\"ip_address\\\":\\\"0.0.0.0\\\",\\\"user_id\\\":\\\"246218764\\\"}\"]}}"
  },
  "fields": {
    "@timestamp": [
      "2018-09-03T08:45:25.296Z"
    ]
  },
  "sort": [
    1535964325296
  ]
}
6

[request][headers][x-user-agent] is an array field so you need to tell Logstash to parse its first element (i.e. [request][headers][x-user-agent][0]).

1 Like
7

That did it!!!!!!!

Thank you so much

8

Hi Magnus,

Any idea how to do this if you have multiple values in the [x-user-agent] field and the length is not quite known?

9

Any idea how to do this if you have multiple values in the [x-user-agent] field and the length is not quite known?

You'd have to use a ruby filter.

10

Hi magnus,

I was able to parse the json object as seperate fields using the ruby filter, but i am not sure how to join the fields. Could you provide some pointers?

Not sure if I should post it here or not. I have posted it in the original thread that I had created.

11

Could you provide some pointers?

Not in this thread.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.