Split arrays of keys and values

Elastic Stack Logstash
1

I'm using Amazon SQS input plugin to get bounced & delivery reports from Amazon SES (Simple Email Service).

One of the fields (mail.headers) is an array of keys and values, like the following:

{
    "name": "Message-ID",
    "value": "<08c903bee6de5daa173f5856a@swift.generated>"
},
{
    "name": "Date",
    "value": "Sat, 24 Jun 2017 22:53:33 +0200"
},
{
    "name": "Subject",
    "value": "Welcome to our website"
},
{
    "name": "From",
    "value": "Example <info@example.com>"
},
{
    "name": "To",
    "value": "email@someone.com"
}

I want to split those fields to be something like

headers.Message-ID: <08c903bee6de5daa173f5856a@swift.generated>
headers.Date: Sat, 24 Jun 2017 22:53:33 +0200
headers.Subject: Welcome to our website
...

I have tried doing the following, but unfortunately, it didn't work as expected and returned only one array ignored the rest:

split {
        add_field => { "headers[%{[mail][headers][name]}]" => "%{[mail][headers][value]}"  }
        field => "[mail][headers]"
}

So, how can I achieve this?

2

The split filter splits multi-line messages into distinct events and it doesn't seems to be what you want to do.

I don't think there is a proper plugin for doing that. You may need to use the ruby plugin with custom code.

3

I was able to split the arrays into key => value using Ruby filter like you said, but unfortunately, what I did moved the events to the top level and I want it to be under headers property

filter {
    if [mail][headers] {
        ruby {
            code => "event.get('[mail][headers]').each {|hash| event.set(hash['name'], hash['value']) }"
        }
    }
}

Output:

{
"Message-ID": "Blah Blah Blah",
"X-Priority": "2 (High)"
}

Expected:

{
    "headers": {
        "Message-ID": "Blah Blah Blah",
        "X-Priority": "2 (High)"
    }
}

Sorry If my question is trivial, but I'm a Ruby developer.

4

Try replacing

event.set(hash['name'], hash['value'])

with

event.set('[headers][' + hash['name'] + ']', hash['value'])
2 Likes
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.