Input file section to read from latest folder

Elastic Stack Logstash
1

Hi

we are looking for solution to read the file from the latest folder in the input path .
our current configuration is as follows,
input {
file {
path => "/var/log/orglogs//G.log"
start_position => "end"
}

}

we have multiple folders in side /var/log/orglogs/ with date as the name of the folder. that will 2016-7-01, 2016-7-02 etc .... ! instead of reading out from we want the logstash configuration to read only from the current date .

Is there a way we can use a parameter to set todays date and pass that parameter to the path , as follows,
latestdate => date +%Y-%-m-%d
path => "/var/log/orglogs/$latestdate/G*.log

PS NOTE:

  1. we can fix the same with logrotate the file on the old folder, but due to the size of the file it takes times and results in delay in reading new file rather logstash reads old file
  2. we tried to create symlink to point to latest directly and it works as well, But we need the date of the folder as field in our grok pattern . When using symlink we get the symlink name instead of the original folder name.

Can some one help ??

Thanks in Advance
Govind

2

Is there a way we can use a parameter to set todays date and pass that parameter to the path , as follows,
latestdate => date +%Y-%-m-%d
path => "/var/log/orglogs/$latestdate/G*.log

I can think of a few options:

  • You can reference environment variables in the Logstash configuration, but then you'll have to restart Logstash at midnight every day.
  • You could regenerate a configuration file at midnight every day.
  • You could point Logstash to a symlink that's maintained by an external script that makes sure it always points to the latest directory. Since you need the path of the input file you can use a ruby filter to extract the path of the real file.
  1. we can fix the same with logrotate the file on the old folder, but due to the size of the file it takes times and results in delay in reading new file rather logstash reads old file

If you rotate the file via a rename (rather the copytruncate) the rotation operation should be quite quick.

3

Yes me too thinking about the second option of rename . Would be great if you can suggest a use case with an example of ruby filter to extract the path.