Logstash File input filter, ingest twice same file name

Elastic Stack Logstash
1

Hi!

Using Logstash File Input Filter, I will ingest text files (actually they will be .gz files) that are "finished" (I mean, they will not be written anymore)
When ingest gets done, file must be automatically deleted (Logstash must delete it)

It works fine, however, when I try to ingest for a second time the same file name, with the same content (I guess the content does not matter) it does nothing, the file does not get processed by Logstash, it does not ingest into Elasticsearch, and it does not get deleted.

Files are copied from a remote server to Logstash server filesystem, so I guess even when file names does not change, inodes does everytime.

I tryed dome input filter configuration but I do not get it to work:

input {
  file {
    id => "inputTest"
    path => "/path/to/files/*.gz"
    mode => read
    #ignore_older => 0
    #sincedb_path => "/dev/null"
	sincedb_path => "/path/to/files/sincedb_test.txt"
  }
}

I played with ignore_older setting it to 0, deleting every time sincedb_test.txt file or its content, but it does not work.
Without sincedb_path and with ignore_older=0 it works once, but to get it to work again I have to delete de ignore_older param, and then set it again.

Which would be the config I'd need to get it working?

Thanks a lot!!

2

Does sincedb_clean_after help? (Make sure you read this thread.)

Setting ignore_older to zero in filebeat turns off age based filtering.

Setting ignore_older to zero in a file input tells it to ignore any files older than zero seconds, so unless filewatch spots the file immediately it will ignore it.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.