Filebeat reads old lines as new every time Dnf writes to its log files

Elastic Stack Beats
1

I have Filebeat monitoring log files on several Linux servers. Recently I noticed that on all my Rocky Linux servers Filebeat reads all lines in files /var/log/dnf.* as new every time the file is changed by Dnf. If I manually add new line to these files only that single line is sent to Logstash as new. I have not seen this issue with any other logfiles nor servers running other Linux distro and same Logtash version. All files under /var/log are being monitored by Filebeat.

Inode and device id for the log files do not change when this happens.
Selinux is not blocking anything when the issue happens.

I have tried fully reinstalling Filebeat and deleting state file from /var/lib/filebeat/registry. These did not help as next time the Dnf log files changed Filebeat started again treating all lines in those files as new.

Filebeat version:
filebeat version 8.14.1 (amd64), libbeat 8.14.1 [c74896ed7acbb92921ee46fa5e3d66d575a8b0a9 built 2024-06-10 22:40:21 +0000 UTC]
(Also happened with 8.14.0. Did not see this issue with 13.x.)

OS version:

NAME="Rocky Linux"
VERSION="8.10 (Green Obsidian)"

The OS was updated to 8.10 from 8.9 around same time as Filebeat was upgraded to 8.14.0.

I have installed Filebeat by adding repository https://artifacts.elastic.co/packages/8.x/yum to the servers.

I have configured Filebeat like this:

 # ============================== Filebeat inputs ===============================
filebeat.inputs:

# --------------------------------- Log input ----------------------------------
- type: journald
  enabled: true
  id: everything
  seek: "cursor"

- type: filestream
  enabled: true
  id: main
  # Make sure no file is defined twice as this can lead to unexpected behaviour.
  paths:
    - /var/log/*.log
    - /var/log/sssd/*.log
    - /var/log/httpd/*.log
  prospector.scanner.exclude_files: ['\.gz$']
  ignore_older: 10m

Filebeat logs show no errors and only this warning appears at startup:

{
  "log.level": "warn",
  "@timestamp": "2024-06-13T05:11:58.602Z",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).setupPipelineLoaderCallback",
    "file.name": "beater/filebeat.go",
    "file.line": 193
  },
  "message": "Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.",
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

As I use Logstash pipeline this should not be problem.

Recent entries in /var/lib/filebeat/registry/filebeat/log.json containing string "dnf.log" look like this:

{"k":"filestream::.global::native::25166348-2050","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[597701186,1718178942],"cursor":null,"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[610146777,1718178942],"cursor":{"offset":40872},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[634027865,1718178942],"cursor":{"offset":83819},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[346556910,1718178943],"cursor":{"offset":116325},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"cursor":{"offset":153860},"meta":{"identifier_name":"native","source":"/var/log/dnf.log"},"ttl":1800000000000,"updated":[793744158,1718178943]}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[520772251,1718178944],"cursor":{"offset":194813},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"cursor":{"offset":234207},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":1800000000000,"updated":[45327220,1718178945]}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[534442749,1718178945],"cursor":{"offset":241516},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[601197313,1718180432],"cursor":{"offset":242635},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"cursor":null,"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":0,"updated":[281470681743360,18446744011573954816]}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":{"offset":246825},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":0}}
{"k":"filestream::main::native::25166348-2050","v":{"cursor":null,"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":1800000000000,"updated":[907490526,1718255518]}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[955452561,1718255518],"cursor":{"offset":25110},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"updated":[932758198,1718255520],"cursor":{"offset":48041},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":1800000000000}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[817349798,1718255521],"cursor":{"offset":77188},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":1800000000000,"updated":[482491627,1718255523],"cursor":{"offset":126269}}}
{"k":"filestream::main::native::25166348-2050","v":{"cursor":{"offset":144104},"meta":{"identifier_name":"native","source":"/var/log/dnf.log"},"ttl":1800000000000,"updated":[201813487,1718255524]}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[201839036,1718255524],"cursor":{"offset":144201},"meta":{"identifier_name":"native","source":"/var/log/dnf.log"}}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[982743532,1718255525],"cursor":{"offset":162164},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[533584943,1718255526],"cursor":{"offset":171096},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[378593998,1718255527],"cursor":{"offset":176456},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"updated":[107644089,1718255528],"cursor":{"offset":250930},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":1800000000000}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[264568838,1718255529],"cursor":{"offset":273665},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[908273025,1718255529],"cursor":{"offset":276025},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}

What could be causing this problem and how could I fix it?

2

Update: I tried this with older server and the same thing happens in Rocky Linux 8.9 too.

3

I have recently migrated to Rocky8 Linux and I have the same exact issue.

4

I still have not been able to figure this out.

I also saw this with some other log files on certain Rocky Linux servers but not others. Duplication on this started during summer but seems to have ended on its own.

Currently I am not seeing this issue on Filebeat 8.15-01 but it is too early to tell if this version does not have the problem or if issue just is not happening right now for some unknown reason.