Logstash output - set output file name

[liavsh (liav)]

Hi

I'm new to logstash...
I'm using logstash to stream logfiles from AWS MSK Kafka to AWS S3 bucket.

I FAIL setting the output filename inside the bucket.
I was able to set the folder that will hold the output file.

here's my logstash.cohfig file

input {
    kafka {
        bootstrap_servers => "${FILEBEAT_KAFKA_HOSTS}"
        topics => ["${TOPIC_PROVIDED_BY_ECS}"]
        tags => "${TOPIC_PROVIDED_BY_ECS}"
    }
}

output {
   s3 {
     bucket => "${DESTINATION_NAME}"
     size_file => "${FILE_SIZE_IN_BYTES}"
     region => "${DESTINATION_CHARACTERISTIC_NAME}"
     prefix => "${TOPIC_PROVIDED_BY_ECS}/"
   }
}

I read in logstash docs that the output file name will consist (among other) from the "tags" value:
section "S3 output file"
in S3 output plugin | Logstash Reference [7.11] | Elastic

I tried using:
tags => "{TOPIC_PROVIDED_BY_ECS}" tags => ["{TOPIC_PROVIDED_BY_ECS}"]
tags => ["justAname"]

But It did not help.

My goal is that the file name will be part of the output file name
E.G when TOPIC_PROVIDED_BY_ECS=PINKFLOYD
Then the file name will be something like:
ls.s3.0de8cfe6290-8fab-4691-910c-c4befed0da5c.2021-02-11T12.30.PINKFLOYD.part81.txt

Prefaerable with more readable date:
ls.s3.0d8cfe6290-8fab-4691-910c-c4befed0da5c.2021-02-11_12-31-16-323.PINKFLOYD.part81.txt

Any Ideas ?

[Badger]

@liavsh

tags => "${TOPIC_PROVIDED_BY_ECS}"

That will add an entry to the [tags] array containing the value of the environment variable TOPIC_PROVIDED_BY_ECS. It seems unlikely that you want that.

If you want to know what topic an event was read from take a look at the decorate_events option on the kafka input. Note that the documentation of that contradicts itself.

If you want the prefix to be set to the topic you would use

prefix => "%{[@metadata][kafka][topic]}"

[liavsh (liav)]

Hi Badger

Thanks a lot for your reply !

When I use the prefix, I'm only able to control the PATH to the output file name and not the output FILE NAME itself. At least , this is what I saw when I was 'playing' with the configuration file.
My goal is to control the out put file name.

[Badger]

@liavsh

My goal is to control the out put file name.

The code does not support that.