Currently, I have the following setup. One machine with installed systemd-journal-upload and another one with systemd-journal-remote. systemd-journal-remote write all received logs into the dedicated directory. Journalbeat read logs from this directory and send them into Logstash-Elastic.
The problem appears when logs are rotated. After rotation, journalbeat starts sending the last message from rotated log again and again, until new rotation happens. After second rotation Journalbeat read all messages from a rotated file and again falls into an infinite loop sending the last message from a rotated log. Records in the registry don't changes although Journalbeat sending events, and according to logs updates checkpoints.
Checkpoint saved to disk. numUpdates=1
OS: "Ubuntu 18.04.1 LTS (Bionic Beaver)"
SYSTEMD: systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid
Journalbeat version 6.7.0 (amd64), libbeat 6.7.0 [14ca49c28a6e10b84b4ea8cdebdc46bd2eab3130 built 2019-03-21 14:39:29 +0000 UTC]
journalbeat.inputs: - paths: ["/var/log/journal/remote"] seek: cursor fields_under_root: true fields: project: xxxx setup.kibana: output.logstash: hosts: ["x.x.x.x:5045"] processors: - add_host_metadata: ~ - add_cloud_metadata: ~ logging.level: debug logging.selectors: ["*"]
Restart helps until rotation.
Is it a problem with my configuration or it's a bug in Journalbeat and I should create an issue?