# GZIP invalid header with Filebeat S3 input and GuardDuty logs
**URL:** https://discuss.elastic.co/t/gzip-invalid-header-with-filebeat-s3-input-and-guardduty-logs/233626
**Category:** Beats
**Tags:** filebeat
**Created:** [May 20, 2020, 11:13pm UTC](https://discuss.elastic.co/t/gzip-invalid-header-with-filebeat-s3-input-and-guardduty-logs/233626 "2020-05-20T23:13:38Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@nhnicwaller](https://discuss.elastic.co/u/nhnicwaller)
#### Post date: [May 20, 2020, 11:13pm UTC](https://discuss.elastic.co/t/gzip-invalid-header-with-filebeat-s3-input-and-guardduty-logs/233626/1 "2020-05-20T23:13:38Z")
I'm trying to ingest logs from AWS GuardDuty using Filebeat, but I'm getting ERROR messages when trying to run the ingest with Filebeat.
```auto
2020-05-20T22:42:27.973Z ERROR [s3] s3/input.go:447 gzip.NewReader failed: gzip: invalid header
2020-05-20T22:42:27.974Z ERROR [s3] s3/input.go:386 createEventsFromS3Info failed for AWSLogs/123456789123/GuardDuty/ca-central-1/2020/05/15/659b5608-a71c-3b42-8979-f851e61d9098.jsonl.gz: gzip.NewReader failed: gzip: invalid header
2020-05-20T22:42:27.974Z WARN [s3] s3/input.go:277 Processing message failed, updating visibility timeout
2020-05-20T22:42:28.005Z ERROR [s3] s3/input.go:447 gzip.NewReader failed: gzip: invalid header
2020-05-20T22:42:28.005Z ERROR [s3] s3/input.go:386 createEventsFromS3Info failed for AWSLogs/123456789123/GuardDuty/ca-central-1/2020/05/14/1557fa6e-f5f8-36ea-add9-28070f1ff7ee.jsonl.gz: gzip.NewReader failed: gzip: invalid header
```
I have configured GuardDuty to [export findings](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_exportfindings.html) to an S3 bucket. The file contents are json-lines/newline-delimited JSON and they are GZIP-compressed. The metadata on the object is as follows:
```auto
Content-Encoding: gzip
Content-Type: application/json
```
Filebeat is configured using the [s3 input plugin](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-s3.html). I'm using Filebeat version 7.6.2. I found the line of code in Filebeat that's [generating this error](https://github.com/elastic/beats/blob/v7.6.2/x-pack/filebeat/input/s3/input.go#L446), but I can't figure out how to work around it.
If I download the file using `aws s3 cp` I can see that the file really is gzip-compressed, and it decompresses just fine on my local computer.
---
### Author:  [@nhnicwaller](https://discuss.elastic.co/u/nhnicwaller)
#### Post date: [May 20, 2020, 11:28pm UTC](https://discuss.elastic.co/t/gzip-invalid-header-with-filebeat-s3-input-and-guardduty-logs/233626/2 "2020-05-20T23:28:45Z")
Could it be that aws-sdk for Go is automatically decompressing the file? I found this issue [#1292 with aws-sdk-go](https://github.com/aws/aws-sdk-go/issues/1292) that says the default transport will decompress the object unless gzip is specified as an accepted encoding.
And filebeat is not specifying gzip as an accepted encoding [when it calls GetObjectRequest()](https://github.com/elastic/beats/blob/v7.6.2/x-pack/filebeat/input/s3/input.go#L401) so maybe filebeat is trying to decompress data that has already been decompressed.
---
### Author:  [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [May 21, 2020, 7:40am UTC](https://discuss.elastic.co/t/gzip-invalid-header-with-filebeat-s3-input-and-guardduty-logs/233626/3 "2020-05-21T07:40:19Z")
You can try to delete the line with "Content-Encoding". The data that filebeat downloads is not valid GZIP content.
---
### Author:  [@nhnicwaller](https://discuss.elastic.co/u/nhnicwaller)
#### Post date: [May 21, 2020, 6:06pm UTC](https://discuss.elastic.co/t/gzip-invalid-header-with-filebeat-s3-input-and-guardduty-logs/233626/4 "2020-05-21T18:06:47Z")
@mtojek Not sure I understand your suggestion?
The object metadata is populated by AWS GuardDuty when it writes objects into the bucket, and Filebeat acts on the new object as soon as it is Put into the bucket because it subscribes to an SQS queue of bucket update notifications.
I could overwrite the metadata for one object, but that doesn't really help with the ingest flow. I can't change the Content-Encoding that is used on new objects created by GuardDuty.
---
### Author:  [@nhnicwaller](https://discuss.elastic.co/u/nhnicwaller)
#### Post date: [May 21, 2020, 7:02pm UTC](https://discuss.elastic.co/t/gzip-invalid-header-with-filebeat-s3-input-and-guardduty-logs/233626/5 "2020-05-21T19:02:17Z")
@mtojek I tested your theory by copying one of the files produced by GuardDuty into the same bucket, but with different metadata. This allowed Filebeat to ingest the file normally, and the events are visible in Kibana.
```auto
aws --profile root s3 cp --metadata Content-Encoding=faketest s3://my-guardduty-bucket/AWSLogs/123456789123/GuardDuty/ca-central-1/2020/05/14/017c8c8f-7ad6-3da5-9c53-8c08ba35b370.jsonl.gz s3://my-guardduty-bucket/AWSLogs/123456789123/GuardDuty/ca-central-1/2020/05/14/017c8c8f-7ad6-3da5-9c53-8c08ba35b370-3.jsonl.gz
```
I did _not_ change the content of the file. So it seems like the data in S3 is valid GZIP content, but Filebeat will fail to process that GZIP content if the the header is `Content-Encoding: gzip`.
I'm still not sure how to make Filebeat work for ingesting GuardDuty logs. Should I have a Lambda that corrupts the Metadata on every object before delivering it to the queue for Filebeat? That does not seem good.
---
### Author:  [@nhnicwaller](https://discuss.elastic.co/u/nhnicwaller)
#### Post date: [May 21, 2020, 8:09pm UTC](https://discuss.elastic.co/t/gzip-invalid-header-with-filebeat-s3-input-and-guardduty-logs/233626/6 "2020-05-21T20:09:34Z")
I'm fairly sure this is a bug in the Filebeat S3 input plugin so I reported a bug on the Issue tracker on GitHub.
>
>
> When using Filebeat with the S3 input plugin (beta), Filebeat will fail processing files that contain JSON lines and are GZIP-compressed....
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 18, 2020, 8:09pm UTC](https://discuss.elastic.co/t/gzip-invalid-header-with-filebeat-s3-input-and-guardduty-logs/233626/7 "2020-06-18T20:09:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.