# How to handle gzip'd gzip'd (twice) files in S3

**URL:** <https://discuss.elastic.co/t/how-to-handle-gzipd-gzipd-twice-files-in-s3/165436>\
**Category:** Logstash\
**Created:** [January 23, 2019, 3:38pm UTC](https://discuss.elastic.co/t/how-to-handle-gzipd-gzipd-twice-files-in-s3/165436 "2019-01-23T15:38:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![markwan](https://avatars.discourse-cdn.com/v4/letter/m/46a35a/32.png) [@markwan](https://discuss.elastic.co/u/markwan)\
**Post date:** [January 23, 2019, 3:38pm UTC](https://discuss.elastic.co/t/how-to-handle-gzipd-gzipd-twice-files-in-s3/165436/1 "2019-01-23T15:38:25Z")

</div>

A client is providing AWS cloud trail logs via an S3 bucket. The S3 input plugin will read .gz files.

However the problem is that the file, eg "cloudtrail\_log\_2019-01-22.gz" is actually gzip'd twice, and should really be named cloudtrail\_log\_2019-01-22.gz.gz"

I have tried the gzip\_lines codec but that assumes each line in a file is gzip'd not the whole file.

Has anyone else seen this problem and found a solution? Right now I'm thinking about a pipeline using s3 input which just unzips and outputs to a file on the local drive, but suffixing with a .gz on the inner gzip file. Then a second pipeline which reads from that using file codec on the new .gz file.

Thoughts? Comments?

---

<div class="post-metadata">

**Author:** ![markwan](https://avatars.discourse-cdn.com/v4/letter/m/46a35a/32.png) [@markwan](https://discuss.elastic.co/u/markwan)\
**Post date:** [January 31, 2019, 9:55am UTC](https://discuss.elastic.co/t/how-to-handle-gzipd-gzipd-twice-files-in-s3/165436/2 "2019-01-31T09:55:21Z")

</div>

Client has changed data. Now the files in S3 are not double gzip'd. However the file does not have a .gz extension. This means s3 input does not automatically recognise data as gzip'd.

Using gzip\_lines codec looks like it's looking for each line to be gzip data, not a whole file that is gzip'd. So data cannot be parsed using that codec.

Is the only solution to extend the s3 input plugin to allow an optional is\_gzip flag that forces gzip behaviour? or is there a simpler solution available?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 28, 2019, 9:55am UTC](https://discuss.elastic.co/t/how-to-handle-gzipd-gzipd-twice-files-in-s3/165436/3 "2019-02-28T09:55:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.