Filebeat read file from beginning always when restart

Elastic Stack Beats
1

I have a problem with filebeat running on Linux, every time I stop the filebeat and then start it, it always read file from the beginning.

The log file uploaded by windows batch every 5 minutes, steps as below:

  1. put temporary file

  2. replace original file by renaming temporary file

Filebeat read the line from last change correctly, except restart.

Linux version: Red Hat Enterprise Linux Server release 6.9 (Santiago)
Filebeat version: 7.16.2

My filebeat configuration

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /logs/AB*
  file_identity.path: ~   
  encoding: GB2312
  parsers:
  - multiline:
      type: pattern
      pattern: '^.*INFO'
      negate: true
      match: after
      flush_pattern: '.*Testing End.*'
      max_lines: 10000

output.kafka:
  hosts: ["192.168.1.1:9092"]
  topic: "filebeat-logs"
  enabled: true
  max_message_bytes: 10000000
  required_acks: 0

Is there any way to stop reading the log files again on restarting filebeat and thus avoiding duplicate logging?

2

Could it be a permission issue? Are you running filebeat as root? It sounds like the user running the filebeat service cannot write to the filebeat registry file to be able to keep track where it left off before shutdown of the service.

What does your logs say or journalctl?

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.