Logstash s3 output

Elastic Stack Logstash
1

My logstash s3 output is not working as defined in the pipeline config. The exact output config is working for other pipelines. The dynamic directory structure(/syslog/%{+YYYY}/%{+MM}/%{+dd}/%{+HH}) defined in the config is not created.
The expected directory structure "s3://my-s3-bucket/syslog/2022/08/12/filename.txt.gz".
What I see in the s3 is "s3://my-s3-bucket/syslog////filename.txt.gz"

output:

output {
  s3 {
    id => "syslog_id"
    bucket => "my-s3-bucket"
    prefix => "/syslog/%{+YYYY}/%{+MM}/%{+dd}/%{+HH}"
    temporary_directory => "/app/syslog_temp"
    encoding => "gzip"
    codec => json_lines
  }
}
2

Have you removed the [@timestamp] field from your events?

3

yes... I see it is repeating field from the syslog header.

mutate {
                remove_field => ["@timestamp"]
        }
4

Those are expanded using the [@timestamp] field. If the field does not exist then they will be empty strings.

1 Like
5

Thank you @Badger. Now the config is working as defined.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.