Filebeat Load All the Entries in Log File Downloaded from Remote Server

Elastic Stack Beats
1

Hi there,

In my scenario, I download log file with wget from remote server to local and parse it with Logstash and Filebeat. For example:

Remote file name: SystemOut.log, and its contents looks like below

[9/26/17 7:05:20:051 EDT] 0000004d SystemOut O [1111121826]
[9/26/17 7:05:20:055 EDT] 0000004d SystemOut O [1111121830]

Filenet sent it to Logstash, Logstash parsed it and stored them into database.

in 10 minutes, I downloaded the file, and it contains below entries:
[9/26/17 7:05:20:051 EDT] 0000004d SystemOut O [1111121826]
[9/26/17 7:05:20:055 EDT] 0000004d SystemOut O [1111121830]
[9/26/17 7:05:20:056 EDT] 0000004d SystemOut O [1111121831]

I found the top 2 entries were parsed and loaded again.

Is there any method to prevent this from happening?

Regards,

Fei

2

Filebeat tracks files by inode, so if you copy over a file it will appear as a new file even if it has the same name. In order to avoid this you will need to instead append to the file or even better install Filebeat where the file is generated..

3

Make sense. I fixed it by overwriting the file.

Thanks, Christian

Fei

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.