How can i make logstash to read file with same filename after processing


I have tried many ways but still not able to get it work
basically, i have logstash to watching one folder and load into ES with read model
input {
file {
path => [
type => "stp"
tags => ["aa"]
mode => "read"
file_completed_action => "delete"
sincedb_clean_after => "2 min"
sincedb_path => "/env/.csvstp"
so first time when i load file aa.csv into the folder it get processed and file been deleted. this is expected

later i put different content into aa.csv and copy to /tm/ folder again, but logstash is not processing unless i restart and clear sincedb file.

I have tried to use sincedb to /dev/null, but it will only process file after restart logstash

is there anyway i can make it

  1. automatically process the file from /tmp/*csv
  2. delete it after process
  3. load the file again if the files with same filename copy to /tmp/ folder

No, it is a known issue that a file that is deleted should be removed from the sincedb, and another known issue that file content fingerprinting would work better than simple inode re-use detection.

even i put the sincedb to /dev/null it not work.. so looks like it cache the position in memory and will not process the next file if it has the same name

The file input always uses an in-memory sincedb. Setting sincedb_path to /dev/null prevents the in-memory sincedb being persisted across restarts.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.