Now I have current .conf file for Logstash (6.3):
input {
file {
path => "/home/ubuntu/mail/attachment/csv/Scan_Report/current.csv"
start_position => "beginning"
stat_interval => 300
#sincedb_path => "/dev/null"
sincedb_path => "/home/ubuntu/mail/attachment/csv/Scan_Report/sincedb"
tags => "CSV_Scan_Report"
}
}
...
And some script (executed after fetchmail has received an email):
rm $HOME/mail/attachment/csv/Scan_Report/current.csv && cp $HOME/mail/attachment/csv/Scan_Report/Scan_Report.csv $HOME/mail/attachment/csv/Scan_Report/current.csv && rm $HOME/mail/attachment/csv/Scan_Report/sincedb
Expected Behavior:
After deleted the sincedb file Logstash reads current.csv from begin (sincedb must be is empty after deleted)
Current Behavior
After deleted the sincedb file Logstash reads current.csv from last seeked position. Logstash create new sincedb file with values from old (deleted) sincedb.
And in Logstash logs file I sees strings:
[2018-11-30T11:22:47,706][DEBUG][logstash.inputs.file ] each: file grew: /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: old size 291055, new size 292834
[2018-11-30T11:22:47,711][DEBUG][logstash.inputs.file ] :modify for /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv, file is not open, opening now
[2018-11-30T11:22:47,712][DEBUG][logstash.inputs.file ] _open_file: /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: opening
[2018-11-30T11:22:47,714][DEBUG][logstash.inputs.file ] /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: sincedb last value 291055, cur size 292834
[2018-11-30T11:22:47,718][DEBUG][logstash.inputs.file ] /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: sincedb: seeking to 291055
sincedb before deleting:
1175841 0 64512 291055
sincedb after deleting and logstash viewed new current.csv file:
1175841 0 64512 292834
And again deleting sincedb, current.csv, send email with new csv file:
sincedb before deleting:
1175841 0 64512 292834
sincedb after deleting and logstash viewed new current.csv file:
1175841 0 64512 292834 (???????)
1176518 0 64512 293519
logs:
[2018-11-30T11:32:47,736][DEBUG][logstash.inputs.file ] each: new inode: /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: old inode was ["1175841", 0, 64512], new is ["1176518", 0, 64512]
[2018-11-30T11:32:47,737][DEBUG][logstash.inputs.file ] :delete for /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv, closing file
[2018-11-30T11:32:47,738][DEBUG][logstash.inputs.file ] _open_file: /home/ubuntu/mail/attachment/csv/Scan_Report/current.csv: opening
If again deleting sincedb, current.csv inod value from current.csv maybe 1175841 and Logstash can't reads all file, only new lines.
Where did Logstash take "sincedb last value" if last sincedb file is deleted? (While .sinnedb_* file is not found in this directory: /usr/share/logstash/data/plugins/inputs/file and /home/ubuntu)
If delete current sincedb file and RESTART Logstash service, then sincedb really is empty...
I need read full file everytime if fetchmail received new email.
P.S. If sincedb_path is set to "/ dev / null", then everything repeats....
P.S.S. Sorry for my ugly english =)