# Filebeat reading logs from S3
**URL:** https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065
**Category:** Beats
**Tags:** filebeat
**Created:** [April 1, 2020, 2:32pm UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065 "2020-04-01T14:32:39Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Nithya](https://discuss.elastic.co/u/Nithya)
#### Post date: [April 1, 2020, 2:32pm UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/1 "2020-04-01T14:32:40Z")
Hi,
I'm trying to get the AWS Logs which is stored in the centralised S3 bucket. I configured the SQS to get the file and push it to the Elastic Cloud index.
I'm facing the below problems:
1. When I see the logs, each line from the log file is storing as a separate doc.
2. Getting a gzip invalid header error while uploading the WAF logs and CloudTrail
ERROR:
**2020-04-01T19:16:31.002+0530|WARN|[s3]|s3/input.go:277|Processing message failed, updating visibility timeout**
**2020-04-01T19:16:31.011+0530|INFO|[s3]|s3/input.go:282|Message visibility timeout updated to 300**
**2020-04-01T19:16:31.035+0530|INFO|[s3]|s3/input.go:282|Message visibility timeout updated to 300**
**2020-04-01T19:16:31.035+0530|ERROR|[s3]|s3/input.go:447|gzip.NewReader failed: gzip: invalid header**
**2020-04-01T19:16:31.035+0530|ERROR|[s3]|s3/input.go:386|createEventsFromS3Info failed for folder/XXXXXXXXXX/waf\_logs/date/filename.gz: gzip.NewReader failed: gzip: invalid header**
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [April 2, 2020, 8:26am UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/2 "2020-04-02T08:26:39Z")
Hey!
Could you share your configuration please?
Also please have a look at the [docs](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-s3.html) and make sure that you don't miss anything like Permissions etc.
Thanks!
---
### Author:  [@Nithya](https://discuss.elastic.co/u/Nithya)
#### Post date: [April 2, 2020, 8:41am UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/3 "2020-04-02T08:41:33Z")
Hi @ChrsMark,
Thanks for your response!
This is my config:
```auto
> filebeat.inputs:
> - type: s3
> queue_url: https://sqs.us-west-2.amazonaws.com/XXXXXXXXXX/sqs-name
> visibility_timeout: 300s
> credential_profile_name: default
> cloud.id: "cloudid"
> cloud.auth: "elastic:{password}"
```
And yes, my AWS profile has admin access.
Thanks!
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [April 2, 2020, 8:44am UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/4 "2020-04-02T08:44:28Z")
Thanks!
Could you share a complete log output of Filebeat too? Please run it in debug mode like `./filebeat -e -d "*"`.
C.
---
### Author:  [@Nithya](https://discuss.elastic.co/u/Nithya)
#### Post date: [April 2, 2020, 10:30am UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/5 "2020-04-02T10:30:23Z")
@ChrsMark
Can we setup a call to discuss on this?
Thanks!
---
### Author:  [@Nithya](https://discuss.elastic.co/u/Nithya)
#### Post date: [April 2, 2020, 10:58am UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/6 "2020-04-02T10:58:17Z")
@ChrsMark Or can you give me a sample config file to get the log from S3 which contains the logs of cloudtrail, cloudfront, vpc flowlogs, cloudwatch and waf logs?
---
### Author:  [@Nithya](https://discuss.elastic.co/u/Nithya)
#### Post date: [April 2, 2020, 12:42pm UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/7 "2020-04-02T12:42:39Z")
@ChrsMark Is there any other module available to collect the logs from S3 Bucket?
Thanks!
---
### Author:  [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)
#### Post date: [April 2, 2020, 4:33pm UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/8 "2020-04-02T16:33:31Z")
@Nithya Thanks for creating this issue here.
```auto
filebeat.inputs:
- type: s3
queue_url: https://sqs.us-west-2.amazonaws.com/XXXXXXXXXX/sqs-name
visibility_timeout: 300s
credential_profile_name: default
expand_event_list_from_field: Records
cloud.id: "cloudid"
cloud.auth: "elastic:{password}"
```
For Cloudtrail logs, they are in json format so `expand_event_list_from_field` is needed for decoding json.
Or you can use the cloudtrail fileset directly in Filebeat. You can run `./filebeat modules enable aws` and then in `modules.d/aws.yml` you should see as section for cloudtrail logs.
---
### Author:  [@Nithya](https://discuss.elastic.co/u/Nithya)
#### Post date: [April 3, 2020, 6:44am UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/9 "2020-04-03T06:44:29Z")
Hi @Kaiyan_Sheng,
How could I read the files which have the content type of application/octet-stream?
Because I'm streaming the CLoudWatch and WAFLogs using Firehose from multiple accounts to a common S3 bucket and it has the content type application/octet-stream.
And what are all the content-type which FileBeat will accept?
Thanks!
---
### Author:  [@Nithya](https://discuss.elastic.co/u/Nithya)
#### Post date: [April 6, 2020, 3:32pm UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/10 "2020-04-06T15:32:55Z")
Hi @Kaiyan_Sheng, @ChrsMark,
Can you check the above comment?
Thanks,
Nithya
---
### Author:  [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)
#### Post date: [April 10, 2020, 6:33pm UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/11 "2020-04-10T18:33:57Z")
@Nithya Sorry for the late response! Right now S3 input in Filebeat reads files with `bufio.NewReader` unless content-type is application/x-gzip, then it uses `gzip.NewReader` instead. There is no special reader for `application/octet-stream` yet.
What error message do you see when you try config below?
```auto
filebeat.inputs:
- type: s3
queue_url: https://sqs.us-west-2.amazonaws.com/XXXXXXXXXX/sqs-name
visibility_timeout: 300s
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 8, 2020, 6:34pm UTC](https://discuss.elastic.co/t/filebeat-reading-logs-from-s3/226065/12 "2020-05-08T18:34:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.