# S3 Filebeat Input Cloudfront Logs - handleSQSMessage failed: json unmarshal sqs message body failed

**URL:** https://discuss.elastic.co/t/s3-filebeat-input-cloudfront-logs-handlesqsmessage-failed-json-unmarshal-sqs-message-body-failed/239716
**Category:** Beats
**Created:** [July 2, 2020, 10:35pm UTC](https://discuss.elastic.co/t/s3-filebeat-input-cloudfront-logs-handlesqsmessage-failed-json-unmarshal-sqs-message-body-failed/239716 "2020-07-02T22:35:02Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![AddChickpeas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/addchickpeas/32/44414_2.png) [@AddChickpeas](https://discuss.elastic.co/u/AddChickpeas)
#### Post date: [July 2, 2020, 10:35pm UTC](https://discuss.elastic.co/t/s3-filebeat-input-cloudfront-logs-handlesqsmessage-failed-json-unmarshal-sqs-message-body-failed/239716/1 "2020-07-02T22:35:02Z")

</div>

Hello,

I'm trying to get the S3 input configured to ingest Cloudfront logs. Problem is, regardless of what settings I use, it runs into a log in can't process and gives me:

```
handleSQSMessage failed: json unmarshal sqs message body failed: invalid character 'd' in literal false (expecting 'a') 

```

The message is always the same so I'm guessing it keeps failing on the same message or one of a similar format.

When this happens, it stops ingesting. Filebeat does not crash or give an error or anything. It just stops and logs are filled with:

```auto
2020-07-02T22:12:01.687Z DEBUG [input] input/input.go:152 Run input                                                                                                                                                     
2020-07-02T22:12:08.656Z WARN [s3] s3/input.go:298 Half of the set visibilityTimeout passed, visibility timeout needs to be updated                                                                                     
2020-07-02T22:12:08.741Z INFO [s3] s3/input.go:305 Message visibility timeout updated to 300 seconds                       

```

Here is my config. I have made it as basic as possible for testing. The same behavior is seen whether I used `expand_event_list_from_field` or not.

```auto
  filebeat.inputs:
      - type: s3
        queue_url: https://sqs.us-east-1.amazonaws.com/328823170987/hotrock-filebeat-cloudfront-logs
        access_key_id: ${AWS_ACCESS_KEY_ID}
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}
        expand_event_list_from_field: Records

```

For questions:

1. Is there any way I can track what message it failed on? I went back to the SQS queue, but the oldest is always a normal, properly formatted JSON with a `Records` as the top level

2. If we can identify the message, is there any way I can filter it before the s3 input tries to parse it as JSON?

3. Any other ideas on what could be causing this?

---

<div class="post-metadata">

### Author: ![AddChickpeas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/addchickpeas/32/44414_2.png) [@AddChickpeas](https://discuss.elastic.co/u/AddChickpeas)
#### Post date: [July 8, 2020, 2:37pm UTC](https://discuss.elastic.co/t/s3-filebeat-input-cloudfront-logs-handlesqsmessage-failed-json-unmarshal-sqs-message-body-failed/239716/2 "2020-07-08T14:37:32Z")

</div>

Going to bump this once. Anybody ran into this issue before/successfully used filebeat for cloudfront logs?

---

<div class="post-metadata">

### Author: ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)
#### Post date: [July 9, 2020, 4:24pm UTC](https://discuss.elastic.co/t/s3-filebeat-input-cloudfront-logs-handlesqsmessage-failed-json-unmarshal-sqs-message-body-failed/239716/3 "2020-07-09T16:24:18Z")

</div>

Hello! Thanks for reporting the issue here. What is the version of Filebeat are you using? Also if you can share with us one Cloudfront log file, that would be very helpful! Thanks!

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 6, 2020, 6:34pm UTC](https://discuss.elastic.co/t/s3-filebeat-input-cloudfront-logs-handlesqsmessage-failed-json-unmarshal-sqs-message-body-failed/239716/4 "2020-08-06T18:34:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
