S3 input plugin is not able to handle encrypted data

Elastic Stack Logstash
2

From your description, I'm not quite sure how the encoding and encryption are layered, but if you have:

  • newline-delimited log messages
  • that have been gzipped
  • and are stored in an encrypted state in S3

then you may find the gzip_lines codec useful.

input {
  s3 {
    access_key_id => "my_access_key_id"
    secret_access_key => "my_secret_access_key"
    bucket => "my_bucket_name"
    region => "myregion"
    interval => 3
    codec => gzip_lines {
      charset => "UTF-8" # your source charset
    }
  }
}

The S3 Input acquires and decrypts the files, handing off chunks of gzipped bytes to the codec, which creates Events. By using the GZip Lines Codec, we are able to decompress the gzip-encoded data into plaintext.