How to handle rotated logs

Elastic Stack Logstash
1

I am analyzing logs using logstash that are getting updated realtime from an application. The application has an inbuilt log rotation process like this. My application always writes to fsl.stream and once this fsl.stream becomes 2GB in size, application renames it to
"fsl.stream.<unix_time_stamp>" with current system's unix timestamp suffix and start writing a fresh fsl.stream

My input configuration is

input {
file {
path => "/usr/logs/fsl.stream.*"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

In the user logs directory here is the live status :

-rw-r--r-- 1 fsl fsl 865266934 Sep 3 13:24 fsl.stream
-rw-r--r-- 1 fsl fsl 2097184834 Aug 27 19:52 fsl.stream.1598538174
-rw-r--r-- 1 fsl fsl 2097207483 Sep 1 20:33 fsl.stream.1598972633

My question : In case of log rotate from fsl.stream to fsl.stream.<unix_time_stamp> Does logstash keep on reading the renamed file until it's end and then switch to new fsl.stream?

Please note that i have no control of logfile rotation policy from the application side and also can not move the logs to another directory for logstash processing as it will introduce delay in processing.

2

logstash will not read fsl.stream since it does not match that pattern. It will read each of the rotated files and continue to tail them in case any additional lines are written to them.

3

Sorry it was a type. The correct syntax i use is :

path => "/usr/logs/fsl.strea*"

Please confirm if the behavior that i am expecting would work?
Will logstash continue to read the file fsl.stream even after rotation to fsl.stream.<unix_time_stamp> and at end of this file, it will switch back to fsl.stream which was created afresh by application.

4

Yes, it should read the new fsl.stream after it is rotated.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.