How can i make logstash to read file with same filename after processing

Elastic Stack Logstash
1

Hi,

I have tried many ways but still not able to get it work
basically, i have logstash to watching one folder and load into ES with read model
input {
file {
path => [
"/tmp/*csv"
]
type => "stp"
tags => ["aa"]
mode => "read"
file_completed_action => "delete"
sincedb_clean_after => "2 min"
sincedb_path => "/env/.csvstp"
}
}
so first time when i load file aa.csv into the folder it get processed and file been deleted. this is expected

later i put different content into aa.csv and copy to /tm/ folder again, but logstash is not processing unless i restart and clear sincedb file.

I have tried to use sincedb to /dev/null, but it will only process file after restart logstash

is there anyway i can make it

  1. automatically process the file from /tmp/*csv
  2. delete it after process
  3. load the file again if the files with same filename copy to /tmp/ folder
2

No, it is a known issue that a file that is deleted should be removed from the sincedb, and another known issue that file content fingerprinting would work better than simple inode re-use detection.

3

thanks..
even i put the sincedb to /dev/null it not work.. so looks like it cache the position in memory and will not process the next file if it has the same name

4

The file input always uses an in-memory sincedb. Setting sincedb_path to /dev/null prevents the in-memory sincedb being persisted across restarts.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.