JSON Array To Object

Elastic Stack Logstash
1

Hi -

Each record of input to my Logstash instance is a JSON array of key-value pairs. I need to translate this array to a single object with the value of the "key" as the field name and the value of the "value" as the field value. For example, this is a single record of input:

[
	{"Name":"FirstName","Value":"Jamie"},
	{"Name":"LastName","Value":"Smith"},
	{"Name":"Email","Value":"test@value.com"},
	{"Name":"State","Value":"MO"}
]

And this is the ElasticSearch mapping to which this record needs to fit:

{
	"data-test": {
		"mappings": {
			"_doc": {
				"properties": {
					"FirstName": { "type": "text" },
					"LastName": { "type": "text" },
					"Email": { "type": "text" },
					"State": { "type": "text" }
				}
			}
		}
	}
}

I've been looking at a series of filters (kv, translate, split, etc.) but have not seen an easy way to make this happen reliably.

Any guidance or wisdom would be greatly appreciated!

Thanks -
Brad

2

You need a ruby filter. Something like this should work:

event.get('name-of-array-field').each { |kv|
  event.set(kv['Name'], kv['Value'])
}
3

This looks promising, but so far no luck. I don't have a name on the array, so I'm not sure how to access it via the event getter. Unfortunately the data is what it is, I can't modify it. The top level element of an event is the array, the elements of the array are the fields themselves. Is there a way for the event object to return the elements of the top-level array without the array itself having a name?

Thanks -
Brad

4

Ah, got it! I ended up applying first a json filter, then the ruby filter, like this:

filter {
    json {
        source => "loan_object"
        target => "parsed_object"
        remove_field => "loan_object"
    }
    ruby {
        code => "event.get('parsed_object').each { |kv| event.set(kv['Name'], kv['Value']) }"
    }
}

Thanks much for the assistance!
Brad