Logstash read mode not reading file with same name

Elastic Stack Logstash
1

Hi

I'm using the read mode in logstash (version-7.7.0) to read my log files and delete them once they've been parsed by logstash. The use case is such that I have a log file in which data will be coming in intervals which is not fixed (Can be minutes or hours) and the disk space on the server is low so we want to remove log files which have been consumed by logstash. Below is my logstash conf file

    input {
        file {
            path => "/path/to/log/file/*"
            file_completed_action => "delete"
            mode => "read"
            sincedb_path => "/dev/null"
      }
    }

    filter {
     grok { match => { "message" => "%{TIMESTAMP_ISO8601:createdtime} %{GREEDYDATA:logmessage}" }
     }
     mutate{
             remove_field => ["message"]
           }
     date{
             match => [ "logtime", "YYYY-MM-dd HH:mm:ss.SSSSSS" ]
             target => "logtime"
     }
    }

    output {
      kafka {
        bootstrap_servers => "kafka1:9092,kafka2:9092"
        topic_id => ["mytopic"]
        codec => "json"
        compression_type => "gzip"
      }
      stdout { codec => rubydebug }
    }

The issue that i'm facing is that logstash initially parses the file and deletes it after consumption but when the file with the same name is created again, logstash is not consuming it. When I change the name then logstash parses the file with the new name and deletes it as well. Please help me out here.

2

This is most likely due to inode reuse. There are multiple open issues related to it. Everyone knows how to fix it, but nobody is working on it.

3

Thanks Badger.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.