logstash-5.2.1 on Ubuntu 16.04
s3 {
access_key_id => "XXXXXXXXXXXXXXXX"
secret_access_key => "XXXXXXXXXXXXXXXXX"
region => "us-west-2"
bucket => "some-bucket"
time_file => 5
size_file => 1000000000000
prefix => "logstash-test2/%{event}/%{host}.%{+YYYY-MM-dd-HH-00-00}"
canned_acl => "private"
codec => "json_lines"
}
There seems to be no way to configure the file names. They are automatically generated. Look at this example:
├── event-type1/
│ ├── logstash-node-001.2017-02-16-00-00-00/
│ │ ├── ls.s3.1045715e-6536-4013-bab6-7be280338c68.2017-02-17T00.44.part2.txt
│ │ ├── ls.s3.41e8091b-bcfe-4ea5-9efb-8fb38a0a06f4.2017-02-17T01.04.part6.txt
│ │ ├── ls.s3.423adab9-50d0-4c14-b978-cc469b2ec9c8.2017-02-17T00.54.part4.txt
│ │ ├── ls.s3.4e2727b0-adc6-4b8d-b8f6-b7687fa00859.2017-02-17T00.49.part3.txt
│ │ ├── ls.s3.50b0d780-1ad2-4370-8fdc-2e324f472d3f.2017-02-17T01.09.part7.txt
│ │ ├── ls.s3.57c7134e-f9db-4a81-b3dc-a84c1a1ee847.2017-02-17T01.14.part8.txt
│ │ ├── ls.s3.6ded936a-ce15-4e42-94a8-145ef8cdac0d.2017-02-17T00.59.part5.txt
│ │ ├── ls.s3.e67e2489-24ae-483f-adeb-514b45dcf1bd.2017-02-17T00.34.part0.txt
│ │ └── ls.s3.e7c757bd-90e9-4f26-a856-cd0ba1999a34.2017-02-17T00.39.part1.txt
│ └── logstash-node-001.2017-02-16-01-00-00/
│ ├── ls.s3.3edf1d89-1b38-42eb-81f0-6df392909407.2017-02-17T01.23.part1.txt
│ ├── ls.s3.5f73dd04-5357-45f1-b49e-85099261834e.2017-02-17T01.28.part2.txt
│ └── ls.s3.64fff5d7-dfa2-4ab1-a976-31ec801dd87f.2017-02-17T01.18.part0.txt
├── event-type2/
│ ├── logstash-node-001.2017-02-16-00-00-00/
│ │ ├── ls.s3.0ff631a1-45fb-413f-8d9a-9ca6e51be459.2017-02-17T00.46.part3.txt
│ │ ├── ls.s3.25931c87-231d-4cec-b7f2-551d9bce6932.2017-02-17T01.01.part6.txt
│ │ ├── ls.s3.4050430f-8c45-444d-9c09-9d4985e71337.2017-02-17T00.31.part0.txt
│ │ ├── ls.s3.481fc4f5-9b69-4d82-b752-fab53ddf38ef.2017-02-17T00.36.part1.txt
│ │ ├── ls.s3.48c346b9-358c-40a0-ace7-66d48456fb65.2017-02-17T01.11.part8.txt
│ │ ├── ls.s3.79f344c0-809c-422b-80d0-4d21a15d2fcf.2017-02-17T01.06.part7.txt
│ │ ├── ls.s3.c659400d-fa49-4766-9717-f5da193c0bf3.2017-02-17T00.51.part4.txt
│ │ ├── ls.s3.d3efcb62-adbb-4f59-bf2b-302e06d908da.2017-02-17T00.56.part5.txt
│ │ └── ls.s3.f4c192cd-f11e-4ba7-b88c-9eb28fca088d.2017-02-17T00.41.part2.txt
│ └── logstash-node-001.2017-02-16-01-00-00/
│ ├── ls.s3.24979abb-21bf-4c2c-8c2f-3af35f568250.2017-02-17T01.21.part1.txt
│ ├── ls.s3.38699922-7e64-4dd8-b321-6e74253f3cc5.2017-02-17T01.26.part2.txt
│ ├── ls.s3.5f9e42fb-037f-447d-83f5-c1ba793ae11c.2017-02-17T01.31.part3.txt
│ └── ls.s3.fef9122f-aa6e-4540-a90f-573c27655684.2017-02-17T01.16.part0.txt
Those file names are hard to parse and sort. I'd love to be able to define either the beginning, or the end (or both) of those filenames. The UUID and the rest can stay in there, but please allow the user to define at least a chunk of that name.
What would be really great would be string interpolation for the user-definable part of those file names. E.g., allow the user to define part of the file name from field values such as %{event} and so on.