S3 output uploading failed BadDigest using gzip encoding

[tomleb]

Hi,

I am trying to setup a logstash output using the s3 output plugin.

I am getting the following error:

[2019-09-27T13:55:57,651][DEBUG][logstash.outputs.s3      ] Periodic check for stale files
[2019-09-27T13:56:05,096][DEBUG][logstash.outputs.s3      ] Rotate file {:strategy=>"LogStash::Outputs::S3::SizeAndTimeRotationPolicy", :key=>"...", :path=>"..."}
[2019-09-27T13:56:05,101][DEBUG][logstash.outputs.s3      ] Queue for upload {:path=>"..."}
[2019-09-27T13:56:08,519][ERROR][logstash.outputs.s3 ] Uploading failed, retrying. {:exception=>Aws::S3::Errors::BadDigest, :message=>"The Content-MD5 you specified did not match what we received.", :path=>"...", :backtrace=>[...]}

The configuration I am using is shown below. If I remove encoding => "gzip" then I am able to upload without any issue.

output {
  s3 {
    access_key_id                       => "${AWS_ACCESS_KEY_ID}"
    secret_access_key                   => "${AWS_SECRET_ACCESS_KEY}"
    bucket                              => "${BUCKET_NAME}"
    codec                               => "json"
    tags                                => "${BUCKET_NAME}"
    temporary_directory                 => "/tmp/logstash/${BUCKET_NAME}"
    prefix                              => "%{+YYYY}/%{+MM}/%{+ww}/%{+dd}"
    validate_credentials_on_root_bucket => false
    additional_settings => {
      "force_path_style" => true
      "follow_redirects" => false
    }
    size_file                           => 52428800
    rotation_strategy                   => "size_and_time"
    encoding                            => "gzip"
  }
}

I couldn't find anything regarding this error anywhere specifically for logstash.

Note: I am not using AWS S3.

Any ideas on where to look?

[tomleb]

I was able to found the root cause after investigating.

Uploading files to my S3 fails when it is not using multipart (not sure yet if it's my S3 or aws-sdk used).

Turns out, when using gzip encoding, the file would be smaller than 15MB so it would upload without using multipart API. I was able to force it by decreasing the upload_multipart_threshold configuration variable.