Logstash output - set output file name

Hi

I'm new to logstash...
I'm using logstash to stream logfiles from AWS MSK Kafka to AWS S3 bucket.

I FAIL setting the output filename inside the bucket.
I was able to set the folder that will hold the output file.

here's my logstash.cohfig file

input {
    kafka {
        bootstrap_servers => "${FILEBEAT_KAFKA_HOSTS}"
        topics => ["${TOPIC_PROVIDED_BY_ECS}"]
        tags => "${TOPIC_PROVIDED_BY_ECS}"
    }
}

output {
   s3 {
     bucket => "${DESTINATION_NAME}"
     size_file => "${FILE_SIZE_IN_BYTES}"
     region => "${DESTINATION_CHARACTERISTIC_NAME}"
     prefix => "${TOPIC_PROVIDED_BY_ECS}/"
   }
}

I read in logstash docs that the output file name will consist (among other) from the "tags" value:
section "S3 output file"
in S3 output plugin | Logstash Reference [7.11] | Elastic

I tried using:
tags => "{TOPIC_PROVIDED_BY_ECS}" tags => ["{TOPIC_PROVIDED_BY_ECS}"]
tags => ["justAname"]

But It did not help.

My goal is that the file name will be part of the output file name
E.G when TOPIC_PROVIDED_BY_ECS=PINKFLOYD
Then the file name will be something like:
ls.s3.0de8cfe6290-8fab-4691-910c-c4befed0da5c.2021-02-11T12.30.PINKFLOYD.part81.txt

Prefaerable with more readable date:
ls.s3.0d8cfe6290-8fab-4691-910c-c4befed0da5c.2021-02-11_12-31-16-323.PINKFLOYD.part81.txt

Any Ideas ?

That will add an entry to the [tags] array containing the value of the environment variable TOPIC_PROVIDED_BY_ECS. It seems unlikely that you want that.

If you want to know what topic an event was read from take a look at the decorate_events option on the kafka input. Note that the documentation of that contradicts itself.

If you want the prefix to be set to the topic you would use

prefix => "%{[@metadata][kafka][topic]}"

Hi Badger

Thanks a lot for your reply :slight_smile: !

When I use the prefix, I'm only able to control the PATH to the output file name and not the output FILE NAME itself. At least , this is what I saw when I was 'playing' with the configuration file.
My goal is to control the out put file name.

The code does not support that.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.