# S3 output uploading failed BadDigest using gzip encoding

**URL:** <https://discuss.elastic.co/t/s3-output-uploading-failed-baddigest-using-gzip-encoding/201433>\
**Category:** Logstash\
**Created:** [September 27, 2019, 4:06pm UTC](https://discuss.elastic.co/t/s3-output-uploading-failed-baddigest-using-gzip-encoding/201433 "2019-09-27T16:06:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomleb](https://avatars.discourse-cdn.com/v4/letter/t/3be4f8/32.png) [@tomleb](https://discuss.elastic.co/u/tomleb)\
**Post date:** [September 27, 2019, 4:06pm UTC](https://discuss.elastic.co/t/s3-output-uploading-failed-baddigest-using-gzip-encoding/201433/1 "2019-09-27T16:06:59Z")

</div>

Hi,

I am trying to setup a logstash output using the s3 output plugin.

I am getting the following error:

```
[2019-09-27T13:55:57,651][DEBUG][logstash.outputs.s3] Periodic check for stale files
[2019-09-27T13:56:05,096][DEBUG][logstash.outputs.s3] Rotate file {:strategy=>"LogStash::Outputs::S3::SizeAndTimeRotationPolicy", :key=>"...", :path=>"..."}
[2019-09-27T13:56:05,101][DEBUG][logstash.outputs.s3] Queue for upload {:path=>"..."}
[2019-09-27T13:56:08,519][ERROR][logstash.outputs.s3] Uploading failed, retrying. {:exception=>Aws::S3::Errors::BadDigest, :message=>"The Content-MD5 you specified did not match what we received.", :path=>"...", :backtrace=>[...]}

```

The configuration I am using is shown below. If I remove `encoding => "gzip"` then I am able to upload without any issue.

```
output {
  s3 {
    access_key_id => "${AWS_ACCESS_KEY_ID}"
    secret_access_key => "${AWS_SECRET_ACCESS_KEY}"
    bucket => "${BUCKET_NAME}"
    codec => "json"
    tags => "${BUCKET_NAME}"
    temporary_directory => "/tmp/logstash/${BUCKET_NAME}"
    prefix => "%{+YYYY}/%{+MM}/%{+ww}/%{+dd}"
    validate_credentials_on_root_bucket => false
    additional_settings => {
      "force_path_style" => true
      "follow_redirects" => false
    }
    size_file => 52428800
    rotation_strategy => "size_and_time"
    encoding => "gzip"
  }
}

```

I couldn't find anything regarding this error anywhere specifically for logstash.

Note: I am not using AWS S3.

Any ideas on where to look?

---

<div class="post-metadata">

**Author:** ![tomleb](https://avatars.discourse-cdn.com/v4/letter/t/3be4f8/32.png) [@tomleb](https://discuss.elastic.co/u/tomleb)\
**Post date:** [October 7, 2019, 6:21pm UTC](https://discuss.elastic.co/t/s3-output-uploading-failed-baddigest-using-gzip-encoding/201433/2 "2019-10-07T18:21:06Z")

</div>

I was able to found the root cause after investigating.

Uploading files to my S3 fails when it is not using multipart (not sure yet if it's my S3 or aws-sdk used).

Turns out, when using `gzip` encoding, the file would be smaller than 15MB so it would upload without using multipart API. I was able to force it by decreasing the `upload_multipart_threshold` configuration variable.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2019, 6:21pm UTC](https://discuss.elastic.co/t/s3-output-uploading-failed-baddigest-using-gzip-encoding/201433/3 "2019-11-04T18:21:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.