Parsing json array with logstash

Elastic Stack Logstash
1

Our logs have http headers as a json array as shown below

"httpRequest": {
      "httpVersion": "HTTP/1.1",
      "args": "",
      "country": "US",
      "requestId": "1-5fec0c53-78cbd10c4248df7666a62d78",
      "clientIp": "107.xx.yy.zz",
      "uri": "/",
      "httpMethod": "GET",
      "headers": [
        {
          "name": "Host",
          "value": "www.xyz.com"
        },
        {
          "name": "User-Agent",
          "value": "some user agent"
        },
        {
          "name": "Accept",
          "value": "*/*"
        },
        {
          "name": "Accept-Encoding",
          "value": "identity,gzip,deflate"
        }
      ]
    }

In our conf file we have tried this:

filter {
    json {
         source => "message"
         }
   split {
          field => "[httpRequest][headers]"
          }

It works but only 1 header pair is parsed and stored in elastic. I think this is due to the fact that the array keys are not unique for each pair. They are same ("name" and "value") and hence only 1 pair is stored and others are probably being overwritten.

Can this be addressed using filter plugin?

2

How do you want to address it? Does this help?

1 Like
3

@Badger thanks for the reference. That is exactly what I was looking for and the filter did the trick. The only issue was that the syntax on that reference thread was incorrect. It should be [Request][HeadersFlattened][#{name}]. It was missing square brackets. Other than that it worked perfectly. Thanks.

4

Oh, yes. logstash used to allow ambiguous field references like "[a]b" but for a couple of years it has insisted upon "[a][b]"

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.