Logstash file Input Plugin to read all files from a directory where log files gets rotated every hour

[shadu88 (Shadab)]

Hello Dear ELkan's
Hope all are doing good!!.
here are details of my query:

1. Logstash version7.10 on Centos7.8
2. Using File input plugin to read files and forward logs to other components
3. Issue Statement : How can i read all file from a directory where .log files are getting rotated every hour without duplicating the data and loose any logs.
4. File pattern based on time eg. mainlog.log-2021042000
5. my configurations:
   input {
   file {
   path => "/data/mypath/tologs/mainlog.log"
   start_position => "beginning"
   sincedb_path => "/etc/logstash/sincedb/null"
   }
   }
6. every hour new file is created
   mainlog.log-2021042000
   mainlog.log-2021042001
   mainlog.log-2021042002
   mainlog.log-2021042003
   mainlog.log-2021042004
   mainlog.log-2021042005
   mainlog.log-2021042006
   mainlog.log-2021042007
   mainlog.log-2021042008
   mainlog.log-2021042009
   mainlog.log-2021042010
   mainlog.log-2021042011
   mainlog.log-2021042012
   mainlog.log-2021042013
   mainlog.log-2021042014
   mainlog.log-2021042015
   mainlog.log-2021042016
   mainlog.log-2021042017
   mainlog.log-2021042018
   mainlog.log-2021042019
   mainlog.log-2021042020
   mainlog.log-2021042021
   mainlog.log-2021042022
   mainlog.log-2021042023

I appreciate your inputs on this thank you in advance.
Stay Safe (y)

[aaron-nimocks]

If you did path => "/data/mypath/tologs/mainlog.log-*" it will read all the files and there should be no duplicates as long as the files don't contain duplicates.

This part is telling it to start over from the beginning each time though. So once you get it working I would remove those and Logstash will maintain which logs and which data has been already ingested so it won't grab those again.

start_position => "beginning"
sincedb_path => "/etc/logstash/sincedb/null"

[shadu88 (Shadab)]

Hello Aaron,

Thanks for quick turnaround.
Absolutely path => "/data/mypath/tologs/mainlog.log-*" will include all my log files, but the catch is same logs gets rotated every hour obviously its a duplicate data.
What's your recommendation for my scenario, where log files rotate every one hour.

Thanks Much!!!

Regards,
Shadab

[aaron-nimocks]

I am not sure if there is a way if your logs are creating duplicate data.

[shadu88 (Shadab)]

Currently my configuration is pointing to active file where logging happens actively.
path => "/data/mypath/tologs/mainlog.log" ( im loosing data by this configuration)
while file rotates to next timestamp file
every hour new file is created
mainlog.log-2021042000
mainlog.log-2021042001
mainlog.log-2021042002 etc etc.