Trouble with Logstash sincedb file


(Amedeo Avogadro) #1

Now I have current .conf file for Logstash (6.3):

input {
	file {
		path => "/home/ubuntu/mail/attachment/csv/Scan_Report/current.csv"
		start_position => "beginning"
		stat_interval => 300
		#sincedb_path => "/dev/null"
		sincedb_path => "/home/ubuntu/mail/attachment/csv/Scan_Report/sincedb"
		tags => "CSV_Scan_Report"
	}
}
...

And some script (executed after fetchmail has received an email):
rm $HOME/mail/attachment/csv/Scan_Report/current.csv && cp $HOME/mail/attachment/csv/Scan_Report/Scan_Report.csv $HOME/mail/attachment/csv/Scan_Report/current.csv && rm $HOME/mail/attachment/csv/Scan_Report/sincedb

Expected Behavior:
After deleted the sincedb file Logstash reads current.csv from begin (sincedb must be is empty after deleted)

Current Behavior
After deleted the sincedb file Logstash reads current.csv from last seeked position. Logstash create new sincedb file with values from old (deleted) sincedb.
And in Logstash logs file I sees strings:
[2018-11-30T11:22:47,706][DEBUG][logstash.inputs.file ] each: file grew: /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: old size 291055, new size 292834
[2018-11-30T11:22:47,711][DEBUG][logstash.inputs.file ] :modify for /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv, file is not open, opening now
[2018-11-30T11:22:47,712][DEBUG][logstash.inputs.file ] _open_file: /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: opening
[2018-11-30T11:22:47,714][DEBUG][logstash.inputs.file ] /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: sincedb last value 291055, cur size 292834
[2018-11-30T11:22:47,718][DEBUG][logstash.inputs.file ] /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: sincedb: seeking to 291055

sincedb before deleting:

1175841 0 64512 291055

sincedb after deleting and logstash viewed new current.csv file:

1175841 0 64512 292834

And again deleting sincedb, current.csv, send email with new csv file:

sincedb before deleting:

1175841 0 64512 292834

sincedb after deleting and logstash viewed new current.csv file:

1175841 0 64512 292834 (???????)
1176518 0 64512 293519

logs:
[2018-11-30T11:32:47,736][DEBUG][logstash.inputs.file ] each: new inode: /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: old inode was ["1175841", 0, 64512], new is ["1176518", 0, 64512]
[2018-11-30T11:32:47,737][DEBUG][logstash.inputs.file ] :delete for /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv, closing file
[2018-11-30T11:32:47,738][DEBUG][logstash.inputs.file ] _open_file: /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: opening

If again deleting sincedb, current.csv inod value from current.csv maybe 1175841 and Logstash can't reads all file, only new lines.

Where did Logstash take "sincedb last value" if last sincedb file is deleted? (While .sinnedb_* file is not found in this directory: /usr/share/logstash/data/plugins/inputs/file and /home/ubuntu)

If delete current sincedb file and RESTART Logstash service, then sincedb really is empty...
I need read full file everytime if fetchmail received new email.

P.S. If sincedb_path is set to "/ dev / null", then everything repeats....
P.S.S. Sorry for my ugly english =)


(Guy Boertje) #2

Firstly, please upgrade the file input to the latest as some important bugs were fixed recently.

bin/logstash-plugin install logstash-input-file --version 4.1.8