Howto Parse JSON into Hash

Elastic Stack Logstash
1

Hello,

I'm working on reconfiguring log messages that come in as JSON strings so that I can POST them to an HTTP service with headers. Using beats input for Logstash. The main hurdle here is that I'd like to convert the "headers" part of the incoming JSON into a Hash that would be appropriate to send down to the http output plugin.

My message format (JSON):

{
	"headers": {
		"header0": "1.1",
		"header1": "",
		"header2": "307622"
	},
	"payload": "data"
}

My configuration (Logstash 5.3.0):

filter {
  json {
    source => ["message"]
  }

  mutate {
    add_field => {"headers" => "%{[headers]}"}
  }
}

This does convert the headers object into this:

logstash_1        |        "headers" => [
logstash_1        |         [0] [
logstash_1        |             [0] "header0",
logstash_1        |             [1] "1.1"
logstash_1        |         ],
logstash_1        |         [1] [
logstash_1        |             [0] "header1",
logstash_1        |             [1] nil
logstash_1        |         ],
logstash_1        |         [2] [
logstash_1        |             [0] "header2",
logstash_1        |             [1] "307622"
logstash_1        |         ],
logstash_1        |         [3] "{\"header1\":\"1.1\",\"header2\":null,\"header3\":\"307622\"}"
logstash_1        |     ]

which is not quite there.

Has anybody here dealt with a similar problem? Could you recommend a alternative configuration?

2

What does message contain?

add_field => {"headers" => "%{[headers]}"}

This doesn't make any sense, remove.

3

Message contains the following JSON:

{
	"headers": {
		"header0": "1.1",
		"header1": "",
		"header2": "307622"
	},
	"payload": "data"
}
4

Made some additional progress. Specifically, I started using the ruby filter plugin with the following configuration:

  ruby {
    init => "require 'json'"
    code => "
      headers_hash = JSON.parse(event.get('headers'))
      event.set('headers_hash', headers_hash)
    "
  }

This extracts all headers as JSON and puts them in a Hashmap "headers_hash".

The issue that remains to be solved is that I can't correctly pass "headers_hash" into the headers field for the http output plugin with the following configuration:

  http {
    http_method => "post"
    headers => headers_hash
    url => "http://localhost:5555/any-route"
    format => "message"
    message => [some_payload]
}

The error reported is:

 22:20:05.766 [LogStash::Runner] ERROR logstash.outputs.http - Invalid setting for http output plugin:

   output {
     http {
       # This setting must be a hash
       # This field must contain an even number of items, got 1
       headers => "headers_hash"
       ...
     }
   }

which is puzzling because "headers_hash" is a Hashmap.

Any ideas?

5

This extracts all headers as JSON and puts them in a Hashmap "headers_hash".

And that's exactly what the json filter does.

which is puzzling because "headers_hash" is a Hashmap.

Yes, but when the headers option is read it doesn't dereference its content, i.e.

   headers => "headers_hash"

means "the string 'headers_hash'", not "the hash found in the field 'headers_hash'".

I don't think it's possible to do what you want. Logstash's configuration language is too limited.

6

Yeah I dug in a bit deeper and confirmed that the headers option does not have the capability to interpolate the values passed into it.

Thank you for following this through.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.