Logstash file Input Plugin to read all files from a directory where log files gets rotated every hour

Elastic Stack Logstash
1

Hello Dear ELkan's
Hope all are doing good!!.
here are details of my query:

  1. Logstash version7.10 on Centos7.8
  2. Using File input plugin to read files and forward logs to other components
  3. Issue Statement : How can i read all file from a directory where .log files are getting rotated every hour without duplicating the data and loose any logs.
  4. File pattern based on time eg. mainlog.log-2021042000
  5. my configurations:
    input {
    file {
    path => "/data/mypath/tologs/mainlog.log"
    start_position => "beginning"
    sincedb_path => "/etc/logstash/sincedb/null"
    }
    }
  6. every hour new file is created
    mainlog.log-2021042000
    mainlog.log-2021042001
    mainlog.log-2021042002
    mainlog.log-2021042003
    mainlog.log-2021042004
    mainlog.log-2021042005
    mainlog.log-2021042006
    mainlog.log-2021042007
    mainlog.log-2021042008
    mainlog.log-2021042009
    mainlog.log-2021042010
    mainlog.log-2021042011
    mainlog.log-2021042012
    mainlog.log-2021042013
    mainlog.log-2021042014
    mainlog.log-2021042015
    mainlog.log-2021042016
    mainlog.log-2021042017
    mainlog.log-2021042018
    mainlog.log-2021042019
    mainlog.log-2021042020
    mainlog.log-2021042021
    mainlog.log-2021042022
    mainlog.log-2021042023

I appreciate your inputs on this thank you in advance.
Stay Safe (y)

2

If you did path => "/data/mypath/tologs/mainlog.log-*" it will read all the files and there should be no duplicates as long as the files don't contain duplicates.

This part is telling it to start over from the beginning each time though. So once you get it working I would remove those and Logstash will maintain which logs and which data has been already ingested so it won't grab those again.

start_position => "beginning"
sincedb_path => "/etc/logstash/sincedb/null"
3

Hello Aaron,

Thanks for quick turnaround.
Absolutely path => "/data/mypath/tologs/mainlog.log-*" will include all my log files, but the catch is same logs gets rotated every hour obviously its a duplicate data.
What's your recommendation for my scenario, where log files rotate every one hour.

Thanks Much!!!

Regards,
Shadab

4

I am not sure if there is a way if your logs are creating duplicate data.

5

Currently my configuration is pointing to active file where logging happens actively.
path => "/data/mypath/tologs/mainlog.log" ( im loosing data by this configuration)
while file rotates to next timestamp file
every hour new file is created
mainlog.log-2021042000
mainlog.log-2021042001
mainlog.log-2021042002 etc etc.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.