Filebeat sends same logs to Logstash on every restart

Elastic Stack Beats
1

Hi, I am using Filebeat 6.5.8 and deployed it in the openshift. The log & registry file available in the mount location. Filebeat sends same logs to Logstash on every restart and when i checked the registry file, can able to see source(source log file, same offset & inode but differene device value) multiple time, which equivalent to restart. That same setup done in my local machine and tried to restart multiple time, it never resend the duplicate logs. could you please help me how filebeat maintains file state [whether source log & offset or FileStateOS{"inode":12832839803,"device":4194522}).

2

Is the registry file stored on persistent storage that can survive a restart?

3

Hope there is no issue registry and location. but the log file details stored in the registry always as in the new line [but referring the old files]. please refer registry content on each filebeat restart.

  1. source:file.log,offset:11772,timestamp:2020-06-02T13:39:26.929784481+01:00,ttl:-1,type:log,meta:null,FileStateOS:{inode:12698398231,device:46}

  2. {source:file.log,offset:11772,timestamp:2020-06-02T13:39:26.94667845+01:00,ttl:-1,type:log,meta:null,FileStateOS:{inode:12698398231,device:1048728}}

  3. {source:file.log,offset:11772,timestamp:2020-06-02T13:39:27.097810676+01:00,ttl:-1,type:log,meta:null,FileStateOS:{inode:12698398231,device:4194522}}

There is no change in the source file, offset but can see the difference in timestamp & filestateos

4

Totally i restarted three times and able to see same logs published three times to Logstash. could you please help me FilestateOs concept and how do we resolve this

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.