Understanding the default behaviour of log rotation of file output plugin

[Satish_Mundru]

Hi,
As I understood that the log is rotated per day by using this config 'path' parameter -
file {
path => "/var/logstash/server-%{+YYYY-MM-dd}.log"
codec => line { format => "%{[message]}"}
}

A) I'm trying to make it a generic in Logstash configuration to pass the filename as event from filebeat, say filename: server-%{+YYYY-MM-DD}.log

when I apply this filename in logstash config as below
file {
path => "/var/logstash/%{filename}"
codec => line { format => "%{[message]}"}
}

file is generated with name server-%{YYYY-MM-DD}.log instead of server-2019-07-08.log.

b) What would be the default behaviour if I give path => "/var/logstash/server.log in config, as noticed it always append the logs. Does it honour the file size. please provide inputs

C) Saw this link to get some details - Logstash Output File Rotate, would not solve my use case.

Please suggest the approach.

Thanks,
Satish

[steffens (Steffen Siering)]

This looks more related to Logstash. Filebeat sends the absolute file path to logstash. You would have to use grok or the ruby filter to find the actual base name (plus/minus dates). Yet I don't know if the file output support a format string to the path setting.

[Badger]

Yeah, the file output will do a sprintf on the path option.

output { file { path => "/tmp/server-%{+YYYY-MM-dd}.log" } }

will result in

Opening file {:path=>"/tmp/server-2019-07-09.log"}

but that pre-supposes that the [@timestamp] field has been parsed correctly out of something in the original message.

[Satish_Mundru]

Thanks Steffens.

i'm new to writing the filters, however will try.
The use cases targeting is filebeat harvesting 5 files and sent to single Logstash, we want to store the corresponding 5 files in logstash centralised location mention in generic path location.

Can anyone help in this regard.

Thanks,
Satish