Journalbeat writes registry file to /registry instead of /var/lib/journalbeat/registry

Elastic Stack Beats
1

I encountered the following error when trying to reconfigure journalbeat to run as non-root, even though the journalbeat user has write permissions to /var/lib/journalbeat:

2019-01-25T09:55:07.494Z	ERROR	instance/beat.go:800	Exiting: open registry: permission denied

With journalbeat 6.5.4 installed from the yum repo / RPM package on Amazon Linux 2, running as a systemd service, and with the default journalbeat.registry_file configuration, it seems like journalbeat incorrectly writes its registry file to /registry (directly in the root directory):

[terom@aws-logstash-dev1 ~]$ sudo ls -la /registry 
-rw------- 1 root root 300 Jan 25 08:39 /registry
[terom@aws-logstash-dev1 ~]$ sudo cat /registry 
update_time: 2019-01-25T08:39:19.00962036Z
journal_entries:
- path: LOCAL_SYSTEM_JOURNAL
  cursor: s=e034fc67d05f457f893a487a8cf261d5;i=37315;b=719c71453c9749b082211972ecbbe4f0;m=375575e0f6;t=580443ff2d762;x=661c04b2840710b1
  realtime_timestamp: 1548405502564194
  monotonic_timestamp: 237656989942

This is a bug, per the docs the default should be relative to the -path.data /var/lib/journalbeat

journalbeat:
  # Name of the registry file. If a relative path is used, it is considered relative to the
  # data path.
  #registry_file: registry

The registry file is not present in the /var/lib/journalbeat directory, but the meta.json is:

[terom@terom-dev1 ~]$ sudo ls -la /var/lib/journalbeat/
total 8
drwxr-x---  2 journalbeat journalbeat   23 Jan 25 08:38 .
drwxr-xr-x 31 root        root        4096 Jan 25 08:23 ..
-rw-------  1 journalbeat journalbeat   48 Jan 25 08:38 meta.json
2

Workaround is to configure the systemd service unit with WorkingDirectory=/var/lib/journalbeat. The journalbeat service now starts as the journalbeat user, and the registry file is correctly written to /var/lib/journalbeat:

[terom@terom-dev1 ~]$ sudo ls -la /var/lib/journalbeat/
total 12
drwxr-x---  2 journalbeat journalbeat   39 Jan 25 10:04 .
drwxr-xr-x 31 root        root        4096 Jan 25 08:23 ..
-rw-------  1 journalbeat journalbeat   48 Jan 25 08:38 meta.json
-rw-------  1 journalbeat journalbeat  301 Jan 25 10:04 registry

# /etc/systemd/system/journalbeat.service
[Unit]
Description=Journalbeat ships systemd journal entries to Elasticsearch or Logstash.
Documentation=https://www.elastic.co/products/beats/journalbeat
Wants=network-online.target
After=network-online.target

[Service]
User=journalbeat
WorkingDirectory=/var/lib/journalbeat
ExecStart=/usr/share/journalbeat/bin/journalbeat -e -c /etc/journalbeat/journalbeat.yml -path.home /usr/share/journalbeat -path.config /etc/journalbeat -path.data /var/lib/journalbeat
Restart=always

[Install]
WantedBy=multi-user.target
3

Unfortunately, it is a bug in 6.x releases in Journalbeat. It is fixed in 7.0: https://github.com/elastic/beats/pull/8973