Hi there,
In my scenario, I download log file with wget from remote server to local and parse it with Logstash and Filebeat. For example:
Remote file name: SystemOut.log, and its contents looks like below
[9/26/17 7:05:20:051 EDT] 0000004d SystemOut O [1111121826]
[9/26/17 7:05:20:055 EDT] 0000004d SystemOut O [1111121830]
Filenet sent it to Logstash, Logstash parsed it and stored them into database.
in 10 minutes, I downloaded the file, and it contains below entries:
[9/26/17 7:05:20:051 EDT] 0000004d SystemOut O [1111121826]
[9/26/17 7:05:20:055 EDT] 0000004d SystemOut O [1111121830]
[9/26/17 7:05:20:056 EDT] 0000004d SystemOut O [1111121831]
I found the top 2 entries were parsed and loaded again.
Is there any method to prevent this from happening?
Regards,
Fei