Filebeat - Sends all the messages on every new message written to log file

[ash_coder (ashwin)]

Hi,

I am running the filebeat with below configuration. Filebeat always publishes all the messages from the log file whenever there is a new message written into it instead of reading the messages after the offset value. I see that offset is being written properly in registry file. Could some one help me to avoid publishing duplicate messages and read only new messages?

  filebeat.inputs:
  - type: log
  enabled: true
  paths:
  - /Users/xxx/loging/logs/*.log
  ignore_older: 24h

  output.file:
  path: "/tmp/filebeat_1"
  filename: filebeat

Registry:
[{"source":"/Users/xxx/loging/logs/1.log","offset":76,"timestamp":"2019-03-06T23:18:36.786319-05:00","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":8620380190,"device":16777220}}]

I cloned the code from the master

[pierhugues (Pier-Hugues Pellerin)]

@ash_coder

I am running the filebeat with below configuration. Filebeat always publishes all the messages from the log file whenever there is a new message written into it instead of reading the messages after the offset value. I see that offset is being written properly in registry file. Could some one help me to avoid publishing duplicate messages a

Hello @ash_coder this is a strange behavior, the only reason why Filebeat would start a file from the beginning is because the inode of the file on disk change. I am assuming that you are currently developing with the master branch.

To verify my theory:

1. Stop filebeat.
2. Get the inode of the file using ls -i myfile.log
3. Start filebeat
4. When filebeat send all the events from the file.
5. Get the inode of the file using ls -i myfile.log

I think the inode has changed.

Are you editing the log file with an editor and adding logs to it manually? If this is the case, some editors on saving will actually create a new file on disk, every new file has a new inode. When this happens Filebeat will read the file completely. If you want to keep the file you need to append to the file like this `echo "hello" >> myfile.log"

You have to append to the file to keep the same inode like

[ash_coder (ashwin)]

@pierhugues

To verify my theory:

1. Stop filebeat.
2. Get the inode of the file using ls -i myfile.log
3. Start filebeat
4. When filebeat send all the events from the file.
5. Get the inode of the file using ls -i myfile.log

Thank you very much @pierhugues. This was it. I was editing my file through vi and that was creating new inode.