Filebeat Lost of first lines when rotating files

Elastic Stack Beats
1

Hello,

I noticed some issues when Filebeat reads a new file: the first line is sometimes skipped. So I made a quick script to test this:

#/bin/bash
for i in {00..100}; do
mv toto.log toto.log.${i} && touch toto.log
echo "test line" >> toto.log
sleep 2
done

I expect to have 101 lines in my Logstash output, but I get a random number of lines. Here is my Filebeat configuration:

path.home: /home/user
path.logs: ${path.home}/logs
path.data: ${path.home}/data
path.config: ${path.home}

logging.level: debug
logging.to_files: true
logging.files:
  path: ${path.logs}
  name: filebeat
  keepfiles: 7
  permissions: 0640



filebeat.inputs:

- type: filestream
  id: test-stream
  enabled: true
  paths:
    - /root/toto*.log
  tags:
    - testing
  ignore_older: 48h
  fields_under_root: true
  scan_frequency: "1s"
  close.on_state_change.renamed: true

output.logstash:
  hosts: ["ipAddress:5044"]
  bulk_max_size: 0

My Filebeat version is 8.16.1. I tried adjusting some parameters, but nothing changed. Does anyone have any idea what's happening and how I can solve my problem?

2

this setting causes Filebeat to stop reading files once they have been renamed

When this option is enabled, Filebeat closes the file handler when a file is renamed. This happens, for example, when rotating files. By default, the harvester stays open and keeps reading the file because the file handler does not depend on the file name. If the close.on_state_change.renamed option is enabled and the file is renamed or moved in such a way that it’s no longer matched by the file patterns specified for the , the file will not be picked up again. Filebeat will not finish reading the file.

Try setting that to false

3

It doesn't seem to have any effect on the result. I still get a random number of lines in my Logstash output.

4

During file rotation, you are renaming the log files to a name that is not part of the glob you've given to filebeat: .log.# vs *.log

5

Also worth noting that these two settings will have a negative performance impact and should likely be removed before moving this into production