# AWS SES Parsing Nested json

**URL:** <https://discuss.elastic.co/t/aws-ses-parsing-nested-json/169423>\
**Category:** Logstash\
**Created:** [February 21, 2019, 1:51pm UTC](https://discuss.elastic.co/t/aws-ses-parsing-nested-json/169423 "2019-02-21T13:51:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![neo](https://avatars.discourse-cdn.com/v4/letter/n/9fc348/32.png) [@neo](https://discuss.elastic.co/u/neo)\
**Post date:** [February 21, 2019, 1:51pm UTC](https://discuss.elastic.co/t/aws-ses-parsing-nested-json/169423/1 "2019-02-21T13:51:32Z")

</div>

Hi Guys, I need your help  
I get logs from S3 buckets SES, the logs change, one for a few, I get field with key value structure,  
I'm trying to parse it but I'm getting errors.

here is my conf file,result and error logs.

conf:  
filter {  
if "emails\_logs" in [tags] {

json {  
source =\> "message"  
target =\> "log\_data"  
}

split { field =\> "[log\_data][Records]" }

json {  
source =\> "[log\_data][Records][Sns][Message]"  
target =\> "[log\_data][Records][Sns][Message]"  
}

mutate { remove\_field =\> ["message"] }

if [log\_data][Records][Sns][Message][mail][headers]  
{  
kv {source =\> "[log\_data][Records][Sns][Message][mail][headers]"  
value\_split =\> ":"  
default\_keys =\> ["name", "not\_exist"]

```
}

```

}  
}  
}

result:

{  
"\_index": "emails\_logs29-2019.02",  
"\_type": "doc",  
"\_id": "5J\_qCWkB\_kihJUD5dbuo",  
"\_version": 1,  
"\_score": null,  
"\_source": {  
"@timestamp": "2019-02-20T07:59:14.240Z",  
"tags": [  
"emails\_logs"  
],  
"@version": "1",  
"log\_data": {  
"Records": {  
"Sns": {  
"SigningCertUrl": "XXX",  
"TopicArn": "XXX",  
"Timestamp": "2019-02-12T12:23:55.725Z",  
"Type": "Notification",  
"UnsubscribeUrl": "XXXX",  
"Signature": "XXX",  
"MessageId": "XXX",  
"SignatureVersion": "1",  
"Subject": null,  
"Message": {  
"delivery": {  
"processingTimeMillis": 670,  
"remoteMtaIp": "00.00.00.00",  
"recipients": [  
"[XXX.com](http://XXX.com)"  
],  
"timestamp": "2019-02-12T12:23:55.671Z",  
"smtpResponse": "00.00.00.00 Message received",  
"reportingMTA": "[XXX.amazonses.com](http://XXX.amazonses.com)"  
},  
"notificationType": "Delivery",  
"mail": {  
"messageId": "XXX-05a1206ad947-000000",  
"headersTruncated": false,  
"sendingAccountId": "99999976013",  
"timestamp": "2019-02-12T12:23:55.001Z",  
"destination": [  
"[XXX.amazonses.com](http://XXX.amazonses.com)"  
],  
"commonHeaders": {  
"subject": "Plain 11",  
"date": "12 Feb 2019 07:23:53 -0500",  
"sender": "[XXX.com](http://XXX.com)\>",  
"replyTo": [  
"[XXX.com](http://XXX.com)"  
],  
"from": [  
"[XXX.com](http://XXX.com)\>"  
],  
"to": [  
"[XXX.amazonses.com](http://XXX.amazonses.com)"  
]  
},  
"sourceIp": "00.00.160.100",  
"headers": [  
{  
"value": "XXX",  
"name": "Received"  
},  
{  
"value": "76",  
"name": "XXXID"  
},  
{  
"value": "1993",  
"name": "XXXontactID"  
},  
{  
"value": "1.0",  
"name": "MIME-Version"  
},  
{  
"value": "[XXX.com](http://XXX.com)\>",  
"name": "Sender"  
},  
{  
"value": "[XXX.com](http://XXX.com)\>",  
"name": "From"  
},  
{  
"value": "[XXX.amazonses.com](http://XXX.amazonses.com)",  
"name": "To"  
},  
{  
"value": "[XXX.com](http://XXX.com)",  
"name": "Reply-To"  
},  
{  
"value": "12 Feb 2019 07:23:53 -0500",  
"name": "Date"  
},  
{  
"value": "Plain 99",  
"name": "Subject"  
},  
{  
"value": "utf-8",  
"name": "Content-Type"  
},  
{  
"value": "base64",  
"name": "Content-Transfer-Encoding"  
}  
],  
"sourceArn": "[XXX.com](http://XXX.com)",  
"source": "[YYY.com](http://YYY.com)"  
}  
},  
"MessageAttributes": {}  
},  
"EventSubscriptionArn": "arn-ffba33ffe044",  
"EventSource": "aws:sns",  
"EventVersion": "1.0"  
}  
}  
},  
"fields": {  
"log\_data.Records.Sns.Message.mail.timestamp": [  
"2019-02-12T12:23:55.001Z"  
],  
"log\_data.Records.Sns.Message.delivery.timestamp": [  
"2019-02-12T12:23:55.671Z"  
],  
"@timestamp": [  
"2019-02-20T07:59:14.240Z"  
],  
"log\_data.Records.Sns.Timestamp": [  
"2019-02-12T12:23:55.725Z"  
]  
},  
"sort": [  
1549974235725,  
1886  
]  
}

logstash error:  
] Failed to create monitoring event {:message=\>"For path: http\_address. Map keys: [:stats, :jvm, :os]", :error=\>"LogStash::Instrument::MetricStore::MetricNotFound"}

---

<div class="post-metadata">

**Author:** ![neo](https://avatars.discourse-cdn.com/v4/letter/n/9fc348/32.png) [@neo](https://discuss.elastic.co/u/neo)\
**Post date:** [March 7, 2019, 12:42pm UTC](https://discuss.elastic.co/t/aws-ses-parsing-nested-json/169423/2 "2019-03-07T12:42:45Z")

</div>

this is the way to do it.

filter {  
if "emails\_logs" in [tags] {

```
            json {
                    source => "message"
                    target => "log_data"
            }
            mutate {
                    remove_field => ["message"]
            }

            split {
                    field => "[log_data][Records]"
            }

            json {
                    source => "[log_data][Records][Sns][Message]"
                    target => "[log_data][Records][Sns][Message]"
            }

            if [log_data][Records][Sns][Message][mail][headers] {
                    ruby {
                            code => "event.get('[log_data][Records][Sns][Message][mail][headers]').each {|hash| event.set('[headers][' + hash['name'] + ']', hash['value']) };
                                     event.remove('[log_data][Records][Sns][Message][mail][headers]')"
                    }
            }
    }

```

}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 4, 2019, 12:42pm UTC](https://discuss.elastic.co/t/aws-ses-parsing-nested-json/169423/3 "2019-04-04T12:42:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.