S3 output plugin - no way to change the file names

Elastic Stack Logstash
1

logstash-5.2.1 on Ubuntu 16.04

s3 {
  access_key_id => "XXXXXXXXXXXXXXXX"
  secret_access_key => "XXXXXXXXXXXXXXXXX"
  region => "us-west-2"
  bucket => "some-bucket"
  time_file => 5
  size_file => 1000000000000
  prefix => "logstash-test2/%{event}/%{host}.%{+YYYY-MM-dd-HH-00-00}"
  canned_acl => "private"
  codec => "json_lines"
}

There seems to be no way to configure the file names. They are automatically generated. Look at this example:

├── event-type1/
│   ├── logstash-node-001.2017-02-16-00-00-00/
│   │   ├── ls.s3.1045715e-6536-4013-bab6-7be280338c68.2017-02-17T00.44.part2.txt
│   │   ├── ls.s3.41e8091b-bcfe-4ea5-9efb-8fb38a0a06f4.2017-02-17T01.04.part6.txt
│   │   ├── ls.s3.423adab9-50d0-4c14-b978-cc469b2ec9c8.2017-02-17T00.54.part4.txt
│   │   ├── ls.s3.4e2727b0-adc6-4b8d-b8f6-b7687fa00859.2017-02-17T00.49.part3.txt
│   │   ├── ls.s3.50b0d780-1ad2-4370-8fdc-2e324f472d3f.2017-02-17T01.09.part7.txt
│   │   ├── ls.s3.57c7134e-f9db-4a81-b3dc-a84c1a1ee847.2017-02-17T01.14.part8.txt
│   │   ├── ls.s3.6ded936a-ce15-4e42-94a8-145ef8cdac0d.2017-02-17T00.59.part5.txt
│   │   ├── ls.s3.e67e2489-24ae-483f-adeb-514b45dcf1bd.2017-02-17T00.34.part0.txt
│   │   └── ls.s3.e7c757bd-90e9-4f26-a856-cd0ba1999a34.2017-02-17T00.39.part1.txt
│   └── logstash-node-001.2017-02-16-01-00-00/
│       ├── ls.s3.3edf1d89-1b38-42eb-81f0-6df392909407.2017-02-17T01.23.part1.txt
│       ├── ls.s3.5f73dd04-5357-45f1-b49e-85099261834e.2017-02-17T01.28.part2.txt
│       └── ls.s3.64fff5d7-dfa2-4ab1-a976-31ec801dd87f.2017-02-17T01.18.part0.txt
├── event-type2/
│   ├── logstash-node-001.2017-02-16-00-00-00/
│   │   ├── ls.s3.0ff631a1-45fb-413f-8d9a-9ca6e51be459.2017-02-17T00.46.part3.txt
│   │   ├── ls.s3.25931c87-231d-4cec-b7f2-551d9bce6932.2017-02-17T01.01.part6.txt
│   │   ├── ls.s3.4050430f-8c45-444d-9c09-9d4985e71337.2017-02-17T00.31.part0.txt
│   │   ├── ls.s3.481fc4f5-9b69-4d82-b752-fab53ddf38ef.2017-02-17T00.36.part1.txt
│   │   ├── ls.s3.48c346b9-358c-40a0-ace7-66d48456fb65.2017-02-17T01.11.part8.txt
│   │   ├── ls.s3.79f344c0-809c-422b-80d0-4d21a15d2fcf.2017-02-17T01.06.part7.txt
│   │   ├── ls.s3.c659400d-fa49-4766-9717-f5da193c0bf3.2017-02-17T00.51.part4.txt
│   │   ├── ls.s3.d3efcb62-adbb-4f59-bf2b-302e06d908da.2017-02-17T00.56.part5.txt
│   │   └── ls.s3.f4c192cd-f11e-4ba7-b88c-9eb28fca088d.2017-02-17T00.41.part2.txt
│   └── logstash-node-001.2017-02-16-01-00-00/
│       ├── ls.s3.24979abb-21bf-4c2c-8c2f-3af35f568250.2017-02-17T01.21.part1.txt
│       ├── ls.s3.38699922-7e64-4dd8-b321-6e74253f3cc5.2017-02-17T01.26.part2.txt
│       ├── ls.s3.5f9e42fb-037f-447d-83f5-c1ba793ae11c.2017-02-17T01.31.part3.txt
│       └── ls.s3.fef9122f-aa6e-4540-a90f-573c27655684.2017-02-17T01.16.part0.txt

Those file names are hard to parse and sort. I'd love to be able to define either the beginning, or the end (or both) of those filenames. The UUID and the rest can stay in there, but please allow the user to define at least a chunk of that name.

What would be really great would be string interpolation for the user-definable part of those file names. E.g., allow the user to define part of the file name from field values such as %{event} and so on.

5 Likes
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.