I have Filebeat monitoring log files on several Linux servers. Recently I noticed that on all my Rocky Linux servers Filebeat reads all lines in files /var/log/dnf.* as new every time the file is changed by Dnf. If I manually add new line to these files only that single line is sent to Logstash as new. I have not seen this issue with any other logfiles nor servers running other Linux distro and same Logtash version. All files under /var/log are being monitored by Filebeat.
Inode and device id for the log files do not change when this happens.
Selinux is not blocking anything when the issue happens.
I have tried fully reinstalling Filebeat and deleting state file from /var/lib/filebeat/registry. These did not help as next time the Dnf log files changed Filebeat started again treating all lines in those files as new.
Filebeat version:
filebeat version 8.14.1 (amd64), libbeat 8.14.1 [c74896ed7acbb92921ee46fa5e3d66d575a8b0a9 built 2024-06-10 22:40:21 +0000 UTC]
(Also happened with 8.14.0. Did not see this issue with 13.x.)
OS version:
NAME="Rocky Linux"
VERSION="8.10 (Green Obsidian)"
The OS was updated to 8.10 from 8.9 around same time as Filebeat was upgraded to 8.14.0.
I have installed Filebeat by adding repository https://artifacts.elastic.co/packages/8.x/yum to the servers.
I have configured Filebeat like this:
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# --------------------------------- Log input ----------------------------------
- type: journald
enabled: true
id: everything
seek: "cursor"
- type: filestream
enabled: true
id: main
# Make sure no file is defined twice as this can lead to unexpected behaviour.
paths:
- /var/log/*.log
- /var/log/sssd/*.log
- /var/log/httpd/*.log
prospector.scanner.exclude_files: ['\.gz$']
ignore_older: 10m
Filebeat logs show no errors and only this warning appears at startup:
{
"log.level": "warn",
"@timestamp": "2024-06-13T05:11:58.602Z",
"log.origin": {
"function": "github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).setupPipelineLoaderCallback",
"file.name": "beater/filebeat.go",
"file.line": 193
},
"message": "Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.",
"service.name": "filebeat",
"ecs.version": "1.6.0"
}
As I use Logstash pipeline this should not be problem.
Recent entries in /var/lib/filebeat/registry/filebeat/log.json containing string "dnf.log" look like this:
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[597701186,1718178942],"cursor":null,"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[610146777,1718178942],"cursor":{"offset":40872},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[634027865,1718178942],"cursor":{"offset":83819},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[346556910,1718178943],"cursor":{"offset":116325},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"cursor":{"offset":153860},"meta":{"identifier_name":"native","source":"/var/log/dnf.log"},"ttl":1800000000000,"updated":[793744158,1718178943]}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[520772251,1718178944],"cursor":{"offset":194813},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"cursor":{"offset":234207},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":1800000000000,"updated":[45327220,1718178945]}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[534442749,1718178945],"cursor":{"offset":241516},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::.global::native::25166348-2050","v":{"ttl":1800000000000,"updated":[601197313,1718180432],"cursor":{"offset":242635},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"cursor":null,"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":0,"updated":[281470681743360,18446744011573954816]}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":{"offset":246825},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":0}}
{"k":"filestream::main::native::25166348-2050","v":{"cursor":null,"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":1800000000000,"updated":[907490526,1718255518]}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[955452561,1718255518],"cursor":{"offset":25110},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"updated":[932758198,1718255520],"cursor":{"offset":48041},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":1800000000000}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[817349798,1718255521],"cursor":{"offset":77188},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":1800000000000,"updated":[482491627,1718255523],"cursor":{"offset":126269}}}
{"k":"filestream::main::native::25166348-2050","v":{"cursor":{"offset":144104},"meta":{"identifier_name":"native","source":"/var/log/dnf.log"},"ttl":1800000000000,"updated":[201813487,1718255524]}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[201839036,1718255524],"cursor":{"offset":144201},"meta":{"identifier_name":"native","source":"/var/log/dnf.log"}}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[982743532,1718255525],"cursor":{"offset":162164},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[533584943,1718255526],"cursor":{"offset":171096},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[378593998,1718255527],"cursor":{"offset":176456},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"updated":[107644089,1718255528],"cursor":{"offset":250930},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"},"ttl":1800000000000}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[264568838,1718255529],"cursor":{"offset":273665},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
{"k":"filestream::main::native::25166348-2050","v":{"ttl":1800000000000,"updated":[908273025,1718255529],"cursor":{"offset":276025},"meta":{"source":"/var/log/dnf.log","identifier_name":"native"}}}
What could be causing this problem and how could I fix it?