# S3 plugin not functioning correctly for GZ files from Firehose

**URL:** <https://discuss.elastic.co/t/s3-plugin-not-functioning-correctly-for-gz-files-from-firehose/190269>\
**Category:** Logstash\
**Created:** [July 12, 2019, 3:37pm UTC](https://discuss.elastic.co/t/s3-plugin-not-functioning-correctly-for-gz-files-from-firehose/190269 "2019-07-12T15:37:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Arpan\_Patnaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arpan_patnaik/32/50010_2.png) [@Arpan\_Patnaik](https://discuss.elastic.co/u/Arpan_Patnaik)\
**Post date:** [July 12, 2019, 3:37pm UTC](https://discuss.elastic.co/t/s3-plugin-not-functioning-correctly-for-gz-files-from-firehose/190269/1 "2019-07-12T15:37:32Z")

</div>

Hello!

I was testing the s3 plugin for a production POC where a Firehose delivery system is delivering Cloudwatch logs into an S3 bucket from where I am reading it with the S3 plugin into logstash

My logstash config is as below:

input {  
s3 {  
bucket =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*"  
region =\> "us-east-1"  
role\_arn =\> "\*\*\*\*\*\*\*\*\*\*\*"  
type =\> "s3"  
interval =\> 10  
additional\_settings =\> {  
"force\_path\_style" =\> true  
"follow\_redirects" =\> false  
}  
}  
}

output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
sniffing =\> false  
index =\> "s3-logs-%{+YYYY-MM-dd}"  
}  
stdout { codec =\> rubydebug }  
}

As I start up logstash locally, I can see the data reaching to logstash but its not in proper format, like below.

{  
"type" =\> "s3",  
"message" =\> "\u001F�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000͒�n\u00131\u0010�\_��\u0015�����x���MC)\u0005D\u0016!\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*",  
"@version" =\> "1",  
"@timestamp" =\> 2019-07-12T15:32:37.328Z  
}

I also tried adding a codec =\> "gzip\_lines" into the configuration, but then logstash was not able to process those files at all. The documentation suggests S3 plugin is supposed to support GZ files out of the box. I was hoping if anyone could point out what I am doing wrong?

Apologies if I didnt post in the right place or with a proper format its my first post here!

Regards,  
Arpan

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 9, 2019, 3:37pm UTC](https://discuss.elastic.co/t/s3-plugin-not-functioning-correctly-for-gz-files-from-firehose/190269/2 "2019-08-09T15:37:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.