Logstash output - set output file name


I'm new to logstash...
I'm using logstash to stream logfiles from AWS MSK Kafka to AWS S3 bucket.

I FAIL setting the output filename inside the bucket.
I was able to set the folder that will hold the output file.

here's my logstash.cohfig file

input {
    kafka {
        bootstrap_servers => "${FILEBEAT_KAFKA_HOSTS}"
        topics => ["${TOPIC_PROVIDED_BY_ECS}"]
        tags => "${TOPIC_PROVIDED_BY_ECS}"

output {
   s3 {
     bucket => "${DESTINATION_NAME}"
     size_file => "${FILE_SIZE_IN_BYTES}"
     prefix => "${TOPIC_PROVIDED_BY_ECS}/"

I read in logstash docs that the output file name will consist (among other) from the "tags" value:
section "S3 output file"
in S3 output plugin | Logstash Reference [7.11] | Elastic

I tried using:
tags => ["justAname"]

But It did not help.

My goal is that the file name will be part of the output file name
Then the file name will be something like:

Prefaerable with more readable date:

Any Ideas ?

That will add an entry to the [tags] array containing the value of the environment variable TOPIC_PROVIDED_BY_ECS. It seems unlikely that you want that.

If you want to know what topic an event was read from take a look at the decorate_events option on the kafka input. Note that the documentation of that contradicts itself.

If you want the prefix to be set to the topic you would use

prefix => "%{[@metadata][kafka][topic]}"

Hi Badger

Thanks a lot for your reply :slight_smile: !

When I use the prefix, I'm only able to control the PATH to the output file name and not the output FILE NAME itself. At least , this is what I saw when I was 'playing' with the configuration file.
My goal is to control the out put file name.

The code does not support that.