logstash-5.2.1 on Ubuntu 16.04
s3 {
access_key_id => "XXXXXXXXXXXXXXXX"
secret_access_key => "XXXXXXXXXXXXXXXXX"
region => "us-west-2"
bucket => "some-bucket"
time_file => 5
size_file => 1000000000000
prefix => "logstash-test2/%{event}/%{host}.%{+YYYY-MM-dd-HH-00-00}"
canned_acl => "private"
codec => "json_lines"
}
The %{event} field is a string, it could take any value from a fixed list of about 25 different kinds of events. So after running for a while you get 25 folders at that level under logstash-test2. At any given time, there's approx. 100 folders in /tmp/logstash, containing various temp files for S3.
When I do "service logstash stop", it takes perhaps 15 ... 20 seconds to complete. All temp files in /tmp/logstash that contain events are pushed to S3 just fine. However, there's a bunch of temp files in there that had zero size when Logstash was shut down, and those are not removed after Logstash exits.
To me that doesn't seem right. If shutdown is clean and all goes well, at the very end /tmp/logstash should be empty. But only non-zero-size files are cleaned up currently.