Journalbeat 6.7 repeats old log records after rotation

Elastic Stack Beats
1

Hello,
Currently, I have the following setup. One machine with installed systemd-journal-upload and another one with systemd-journal-remote. systemd-journal-remote write all received logs into the dedicated directory. Journalbeat read logs from this directory and send them into Logstash-Elastic.

The problem appears when logs are rotated. After rotation, journalbeat starts sending the last message from rotated log again and again, until new rotation happens. After second rotation Journalbeat read all messages from a rotated file and again falls into an infinite loop sending the last message from a rotated log. Records in the registry don't changes although Journalbeat sending events, and according to logs updates checkpoints.

Checkpoint saved to disk. numUpdates=1

System

OS: "Ubuntu 18.04.1 LTS (Bionic Beaver)"

SYSTEMD: systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

Journalbeat version 6.7.0 (amd64), libbeat 6.7.0 [14ca49c28a6e10b84b4ea8cdebdc46bd2eab3130 built 2019-03-21 14:39:29 +0000 UTC]

journalbeat.yml 
journalbeat.inputs:
- paths: ["/var/log/journal/remote"]
  seek: cursor
  fields_under_root: true
  fields:
    project: xxxx

setup.kibana:
output.logstash:
  hosts: ["x.x.x.x:5045"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

logging.level: debug
logging.selectors: ["*"]

Restart helps until rotation.
Is it a problem with my configuration or it's a bug in Journalbeat and I should create an issue?

2

So after some research, I found the following.
This behaviour is unrelated to journal-remote. It was repeated on another ubuntu 18 with pretty default config, in which paths was not specified.

Journalbeat sent record from rotated log every time new record added into a new journal file. It looks for me as journalbeat receive some notify for new events. But uses the wrong source for records.

On version 6.6.2 I can't see this problem, mainly because after rotation journalbeat don't send anything. Possibly related to https://github.com/elastic/beats/issues/9533

3

Created an issue https://github.com/elastic/beats/issues/11758

4

How did you solve this issue I am facing this issue right now,
I am taking logs from journalctl and giving input in logstash, then logstash sent it to kibana.

The last log from journalbeat-7.2 is repeating continuously in kibana, but there is only one entry of that log in journalctl.

journalbeat.inputs:
- paths: ["/var/log/journal/ec2a64c31fceb7b6eee4bd5"]
  seek: cursor
  include_matches:
      - "syslog.identifier=XXXXXXX"
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
        hosts: ["xxx.xxx.xxx.xx:XXXX"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
1 Like
5

Upgraded to 7.3.0 and it fixed my issue , I guess version 6.8.0 and 7.2.0 was broken somewhere.

but with the latest update, it fixed.