# S3 input plugin is not able to handle encrypted data

**URL:** <https://discuss.elastic.co/t/s3-input-plugin-is-not-able-to-handle-encrypted-data/147415>\
**Category:** Logstash\
**Created:** [September 5, 2018, 1:23pm UTC](https://discuss.elastic.co/t/s3-input-plugin-is-not-able-to-handle-encrypted-data/147415 "2018-09-05T13:23:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Satej](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@Satej](https://discuss.elastic.co/u/Satej)\
**Post date:** [September 5, 2018, 1:23pm UTC](https://discuss.elastic.co/t/s3-input-plugin-is-not-able-to-handle-encrypted-data/147415/1 "2018-09-05T13:23:39Z")

</div>

Hello All,  
I have setup the "Kinesis Firehose stream" which accepts the log data from "kinesis data stream" and stores these logs as backup on S3 bucket in gz format.  
For this configuration I have kept the KMS encryption on and its value is set to "aws/s3" which is default one.  
Now, I am trying to parse the KMS encrypted logs which are stored on S3 in gz format by Kinesis firehose, but I am not getting the data in decrypted format. I am using following configuration to fetch the data from S3 bucket,

input{  
s3 {  
"access\_key\_id" =\> "my\_access\_key\_id"  
"secret\_access\_key" =\> "my\_secret\_access\_key"  
"bucket" =\> "my\_bucket\_name"  
"region" =\> "myregion"  
"interval" =\> 3  
}  
}

output{  
stdout { codec =\> rubydebug}  
}

And the output which I am getting on console is as follows,

 ![S3pluginoutput](https://us1.discourse-cdn.com/elastic/original/3X/6/e/6ef5dd2eebaf864b9187628e1f761c8093ae4da1.png)

Is there any additional setting I need in S3 input plugin to get the decrypted data?

Thanks for the help in advance.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [September 5, 2018, 5:51pm UTC](https://discuss.elastic.co/t/s3-input-plugin-is-not-able-to-handle-encrypted-data/147415/2 "2018-09-05T17:51:06Z")

</div>

From your description, I'm not quite sure how the encoding and encryption are layered, but if you have:

- newline-delimited log messages
- that have been gzipped
- and are stored in an encrypted state in S3

then you may find the `gzip_lines` codec useful.

```auto
input {
  s3 {
    access_key_id => "my_access_key_id"
    secret_access_key => "my_secret_access_key"
    bucket => "my_bucket_name"
    region => "myregion"
    interval => 3
    codec => gzip_lines {
      charset => "UTF-8" # your source charset
    }
  }
}

```

The S3 Input acquires and decrypts the files, handing off chunks of gzipped bytes to the codec, which creates Events. By using the GZip Lines Codec, we are able to decompress the gzip-encoded data into plaintext.

---

<div class="post-metadata">

**Author:** ![Satej](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@Satej](https://discuss.elastic.co/u/Satej)\
**Post date:** [September 6, 2018, 6:18am UTC](https://discuss.elastic.co/t/s3-input-plugin-is-not-able-to-handle-encrypted-data/147415/3 "2018-09-06T06:18:14Z")

</div>

Thanks for the reply. I tried setting the codec to gzip\_lines but I am getting following exception,  
 ![CharsetSetting](https://us1.discourse-cdn.com/elastic/original/3X/3/b/3b333bb5d73ae999a29c916c2c4ddf77641a94bb.png)

For input data, we have wired up the cloudwatch logs loggroup to kinesis stream, and this kinesis stream is attached to firehose stream which is storing data on S3.

The sample decoded input string is as follows,

{"messageType":"DATA\_MESSAGE","owner":"123456789","logGroup":"logstash\_test\_loggroup","logStream":"logstash\_test\_loggroup\_logstream","subscriptionFilters":["logstash-test-stream"],"logEvents":[{"id":"34257304609312797092785065151043143816257725575611285504","timestamp":1536150666910,"message":"{"logType":"critical","loglevel":"critical","Method":"GET","Url":"/dummy","Status":200,"Length":15,"CorrelationId":"aa4e93f9-a310-46ce-aff3-54ef94c069ad","LogCreationTime":"2018-9-5 18:01:05"}"}]}

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [September 6, 2018, 8:32pm UTC](https://discuss.elastic.co/t/s3-input-plugin-is-not-able-to-handle-encrypted-data/147415/4 "2018-09-06T20:32:08Z")

</div>

I was wrong. The `gzip_lines` codec is a community-provided codec that requires `data` to be something that responds to `read` (Like an IO or open File), and by default most inputs pass a string to the codec. In its current state, it doesn't appear to work with most inputs.

The S3 Input is documented to automatically decompress `.gz` inputs, so it should be able to decompress without a special codec. Can you run the original again with debug-level logging enabled? This can be done with either the `log.level: debug` setting in your `logstash.yml` configuration, or with the `--log.level debug` command-line flag.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 4, 2018, 8:46pm UTC](https://discuss.elastic.co/t/s3-input-plugin-is-not-able-to-handle-encrypted-data/147415/5 "2018-10-04T20:46:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.