Unpack a list of hashes into new fields

[antonisnyc94]

Hello,

I am having a field data.aws.httpRequest.headers , and it has a list of hashes as seen below.

{
  "name": "host",
  "value": "testwebsite.com"
},
{
  "name": "authorization",
  "value": "token hidden"
},
{
  "name": "sec-ch-ua",
  "value": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"96\", \"Google Chrome\";v=\"96\""
},
{
  "name": "sec-ch-ua-mobile",
  "value": "?0"
},
....

How can I use ruby to unpack the list into new fields, making the name of each hash in the list an additional field with its value as the value?

e.g data.aws.httpRequest.headers.host: testwebsite.com for the first one and so on?

I would greatly appreciate any help!

Thanks in advance,
Tony

[Badger]

This post should give you some ideas.

[antonisnyc94]

Thank you so much @Badger that indeed helped a lot! The only problem is that i want to keep the same name of the field and i cant figure out how to do this seamlessly through ruby. Do you know how to do it? The below code works but changing the field using logstash mutate seems kinda idiotic to me.

                  if [data][aws][httpRequest][headers] {
                    mutate {
                      rename => [ "[data][aws][httpRequest][headers]", "[data][aws][httpRequest][headersTemp]"]
                    }
                    ruby {
                      code => '
                                data = event.get("[data][aws][httpRequest][headersTemp]")
                                data.each_index { |x|

                                    # remove token from authorization header
                                    if data[x]["name"] == "authorization" || data[x]["name"] == "Authorization" || data[x]["name"] == "X-Mandrill-Signature"
                                        data[x]["value"] = "token hidden"
                                    end

                                    #making the maps in the loop into their own field and capitilizing the name
                                    name = data[x]["name"].split(/ |\_|\-/).map(&:capitalize).join("-")
                                    value = data[x]["value"]

                                    event.set("[data][aws][httpRequest][headers][#{name}]", value)
                                }
                                '
                        }

                        mutate {
                        remove_field => [ "[data][aws][httpRequest][headersTemp]" ]
                        }
                        
                      }

[Badger]

In a ruby filter, event.remove deletes a field and also returns its value. So instead of renaming the field you could do

data = event.remove("[data][aws][httpRequest][headers]")

[antonisnyc94]

@Badger

event.remove("[data][aws][httpRequest][headers]")

Thank youu so much! It's wayyy simpler than expected!