After restart, logstash doesn't read only new files

Elastic Stack Logstash
1

Hello, is it possible when we stop and restart logstash doesn't read all the files that match the path from the beginning and read only new files appended?

2

Hi,

You can read this documentation about the sincedb path !

3

Can I use it for production?

4

What do you mean ?

this plugin is a fully developped logstash plugin.

If you have a logstash input you're already using the input "plugin"

5

Hello,
I already use logstash and this is the input that I work on :

input {
  file {
    path => "/var/myapp/*/error.log"
    start_position => "beginning"
    type => "logs"
    sincedb_path => "/dev/null"
 }
}
  • in the path : is a subdirectory, create everyday
    for example : today 1_12_2021
    /var/myapp/1_12_2021/error.log
    after running logstash this file is treated
    at 2_12_2021
    I had 2 subdirectories, each one contain an error file:
    1_12_2021 and 2_12_2021
    I restart logstash it will treat 2 files
    how can I configure logstash to make it read only from the new subdirectory?
6

If you set sincedb_path to /dev/null, this will make logstash read ALL files when you restart because it is not tracking which file was already read.

Remove the sincedb_path from your configuration or change it to point to a file, something like /var/lib/logstash/track-files.db, so logstash can track what files it alread read.

7

I will try it
Thank you

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.