Logstash file read , mode read, sincedb, duplicate events after restart logstash

Elastic Stack Logstash
1

Hi Guys
Can you help me
I have a problem with reading files from logstash.
After restarting logstash, it rereads the files again and sends events to Elastic.

This folder is mounted from another server /opt/Logstore/TMP/*.gz

./logstash --version
Using bundled JDK: /usr/share/logstash/jdk
logstash 7.10.0

Here is my config

input {

    file {
        
       path => ["/opt/Logstore/TMP/*.gz"]
        mode => "read"
        start_position => "beginning"
        discover_interval => 15
        close_older => "1 minute"
        file_completed_action => "log"
        file_completed_log_path => "/etc/logstash/conf.d/filereader/complete_log_k.db"
        sincedb_path => "/etc/logstash/conf.d/filereader/since_k.db"
        sincedb_clean_after => "180"
    }

}

output {

    tcp {
        host => "1.1.1.1"
        port => 3333
        codec => line {
            format => "%{message}"
        }
    }

}
`````````````````
2

There used to be a bug where sincedb_clean_after was actually a number of seconds, not days as documented. However, that was fixed a while back.

I suggest you look at the contents of /etc/logstash/conf.d/filereader/since_k.db and see if it contains the entries you expect after you shut logstash down.

Note that entries are not persisted to disk immediately. If you create /opt/Logstore/TMP/1.gz there will not be an entry on disk. If you then create /opt/Logstore/TMP/2.gz an entry will appear on disk for /opt/Logstore/TMP/1.gz etc. Do not worry about the sincedb on disk while logstash is running, just check it after logstash has stopped.

3

Hi Badger
Thanks for you reply.
I changed sincedb_clean_after => "180" to sincedb_clean_after => "180d"

Problem is that i can see in sincedb my duplicate files and then in Elasticsearch after restart logstash

But, if my folder is local, not shared, logstash is working fine.

I can copy files from shared folder to local, but i think this is not good idea :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.