Resend the data from the beginning of a file with sincedb setting

[SongJieCN]

Hi Guys,

Version: 5.0.1
Operating System: CentOS6.8
Config File (if you have sensitive info, please remove it):
input {
file {
path => [
"/opt/Logs/*",
]
start_position => "beginning"
sincedb_path => "/opt/tmp/.sincedb"
}
output{
}
} # END INPUT

Sample Data:
Steps to Reproduce:
[1] We try to use the configuration to send our data within multi-files
[2] After logstash be interrupted and we restart the logstash sending process
[3] We found redundancy data in our receiver
[4] After check the data, We guess the mechanism is resending from the beginning of file.
Am i right? Or Some config mistakes here?
Thank you very much!

[magnusbaeck (Magnus Bäck)]

Your configuration should not cause previously processed data to be resent. You can try to increase the log level to find out more about what Logstash is doing and why it restarts from the beginning of the file.

[SongJieCN]

Hi Magnusbaeck,

Thanks for your reply.
You mean we can increase the log level? Could you share some samples to me? thx.

And Let me clear my question again.
We have 3 files in the folder, We press multi Ctrl+C to interrupt the process while the second file being transfer.
Then, We restart the Logstash, for the first file, It is not be resent. But the second file whole file was resent.
Actually, We already received part of the second file.

Thanks

[magnusbaeck (Magnus Bäck)]

You mean we can increase the log level?

Yes, there's a logstash.yml option for it as well as a command line option.

We have 3 files in the folder, We press multi Ctrl+C to interrupt the process while the second file being transfer.
Then, We restart the Logstash, for the first file, It is not be resent. But the second file whole file was resent.

If you press Ctrl+C multiple times and don't allow Logstash to shut down gracefully it's possible that it won't write the sincedb file and record how much of the file it has processed. Do you get the same behavior if you press Ctrl+C a single time and wait for Logstash to shut down?