Logstash s3 output doesn't clean up temporary files

Elastic Stack Logstash
1

logstash version: docker 5.1.1
s3 output plugin version: 4.0.3
plugin config:

output {
  s3 {
    bucket => "some_bucket"
    access_key_id => "some_key"
    secret_access_key => "some_secret"
    region => "eu-west-1"
    codec => "json_lines"
    encoding => "none"
    size_file => "16777216"
    time_file => "1"
    upload_workers_count => "8"
    restore => "true"
  }
}

When trying to migrate from an old combination of logstash 2.4.1 + s3 output plugin 3.2.0 to the above described latest versions, the output plugin now seems to fail to clean up after itself. The temporary directories are created inside /tmp/logstash with a file inside them like so:

/tmp/logstash/02880db8-7da0-40e3-a64a-e7899d06153f/ls.s3.49146331-8fc2-48a7-8ef8-5a9854f444fc.2017-01-12T12.23.part27.txt

The files are uploaded to s3, but they remain on disk eventually filling it up.
An excerpt from the logs:

$ cat stdout | grep c79e45ffdc7.2017-01-12T12.41.part59
12:41:50.190 [[main]>worker1] DEBUG logstash.outputs.s3 - Rotate file {:strategy=>"LogStash::Outputs::S3::SizeAndTimeRotationPolicy", :key=>"ls.s3.ca3c5216-d807-4386-8c40-fc79e45ffdc7.2017-01-12T12.41.part59.txt", :path=>"/tmp/logstash/30818e0b-a354-4ab0-be56-8b8e9c29645c/ls.s3.ca3c5216-d807-4386-8c40-fc79e45ffdc7.2017-01-12T12.41.part59.txt"}
12:41:50.190 [[main]>worker1] DEBUG logstash.outputs.s3 - Queue for upload {:path=>"/tmp/logstash/30818e0b-a354-4ab0-be56-8b8e9c29645c/ls.s3.ca3c5216-d807-4386-8c40-fc79e45ffdc7.2017-01-12T12.41.part59.txt"}
12:41:53.538 [S3 output uploader, file: /tmp/logstash/30818e0b-a354-4ab0-be56-8b8e9c29645c/ls.s3.ca3c5216-d807-4386-8c40-fc79e45ffdc7.2017-01-12T12.41.part59.txt] DEBUG logstash.outputs.s3 - Removing temporary file {:file=>"/tmp/logstash/30818e0b-a354-4ab0-be56-8b8e9c29645c/ls.s3.ca3c5216-d807-4386-8c40-fc79e45ffdc7.2017-01-12T12.41.part59.txt"}

Yet the directory and the file inside it remain on disk.

Might be worth to add, that logstash is running in docker with the volume /tmp/logstash is mapped out onto the host's /tmp/logstash. The same configuration, same setup worked flawlessly with the previous versions.

Any ideas?

Thanks,

János

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.