Duplicate messages created by Journalbeat 6.7.1-1

Elastic Stack Beats
1

I'm running journalbeat-6.7.1-1 on CentOS 7.6 and Fedora 28, and i'm noticing that journal beat is duplicating some but NOT all messages. The only thing that appears different between the messages is the journalbeat_read_timestamp field.

Exampe:

Journalctl output shows four (4) messages:

Apr 08 17:20:05 fedora.ddiguru.net systemd[1]: Starting system activity accounting tool...
Apr 08 17:20:05 fedora.ddiguru.net audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 08 17:20:05 fedora.ddiguru.net audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 08 17:20:05 fedora.ddiguru.net systemd[1]: Started system activity accounting tool.

Journalbeat sent the following five (5) messages to my Graylog/Elasticsearch cluster(s):

image
image.png935×462 56.5 KB

My configuration is fairly simple:

#----------------------------------------- Journalbeat Input --------------------------------------------------
journalbeat.inputs:
- paths:
  - "/var/log/journal/9926d0e1c68d4a1e9c9a69fd51bd65d8/system.journal"
  seek: cursor
  cursor_seek_fallback: head

#------------------------------------------- Logstash Output -------------------------------------------------
output.logstash:
  hosts: ["192.168.4.37:5044"]

seccomp.enabled: false

Is there something i'm missing? Do i need to tune this with a memory queue? other? Please advise. Thanks

2

I probably have the same problem. Can you confirm or deny what these duplicate messages appear after journald rotates log?

3

This morning it appeared that logging had stopped as a result of the log rollover. I have a cron job that runs hourly to restart the service that looks something like this:

#!/bin/sh

set -ue

SYSTEM_JOURNAL_FILE_ID=$(journalctl --file /run/log/journal/71da943ecfb04e7a8995f336453a9bf3/system.journal --header | awk -F ': ' '/File ID/ { print $2 }')

if [ ! -e /var/lib/journalbeat/system-journal.id ] || [ "$SYSTEM_JOURNAL_FILE_ID" != "$(cat /var/lib/journalbeat/system-journal.id)" ]; then
  systemctl try-restart journalbeat && echo "$SYSTEM_JOURNAL_FILE_ID" > /var/lib/journalbeat/system-journal.id
fi

This was a script i found while troubleshooting the systemd.log rollover issue/bug.

After restarting promptly at 0700, yes, i'm still getting duplicated messages.

4

I've looked back in time and can ascertain that this issue started in 6.7.0, I viewed all messages that came in from 6.6.2 journalbeats and this problem did not exist.

5

can conform this. 6.7.0 and 6.7.1 send duplicates. with 6.6.2 everything is fine.

6

I am seeing the same issues. I think sometimes with the same read_timestamp as well but mostly this is the only field (besides the id) where the records differ.

image
image.png2560×1360 78.5 KB

image
image.png2560×1360 47.8 KB

7

Same problem in the just released 7.0.0 stack.