journalbeat 6.7.1 tarball
journalbeat is run via systemd unit with user/group journalbeat/journalbeat
# ps -C journalbeat o lstart,pid,user:14,group:14,cmd STARTED PID USER GROUP CMD Tue May 21 13:33:36 2019 1524 journalbeat journalbeat /opt/journalbeat/current/journalbeat run --path.config /opt/journalbeat/ --c journalbeat.yml
Everything is working fine.
I noticed that our provisioning/config-management was "fixing" file ownership in the journalbeat install every few minutes. This turns out to be the journalbeat registry file which is somehow getting ownership set to
changed ownership of '/opt/journalbeat/journalbeat-6-7-1/data/registry' from root:root to journalbeat:journalbeat
I used auditd to confirm journalbeat was making the change. You can see below that journalbeat is deleting the old registry file and creating a new one with uid/gid 0 (root).
You can also see in the SYSCALL line that the uid/gid of the journalbeat process is 0 (root)...which I don't really understand since it should be running as 995 (journalbeat).
Also the PID in the audit log SYSCALL line doesn't match the PID from
ps...each registry relevant SYSCALL in the audit (I only show one) has a different PID. Not sure if that is relevant.
# auditctl -w /opt/journalbeat/journalbeat-6-7-1/data/registry -p wax # grep journalbeat /var/log/audit/audit.log | grep -v CONFIG_CHANGE type=SYSCALL msg=audit(1558701993.212:1178): arch=c000003e syscall=264 success=yes exit=0 a0=ffffffffffffff9c a1=c4200446f0 a2=ffffffffffffff9c a3=c420044720 items=5 ppid=9618 pid=8121 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="journalbeat" exe="/opt/journalbeat/journalbeat-6-7-1/journalbeat" key=(null) type=CWD msg=audit(1558701993.212:1178): cwd="/opt/journalbeat/journalbeat-6-7-1" type=PATH msg=audit(1558701993.212:1178): item=0 name="/opt/journalbeat/current/data/" inode=385637 dev=08:01 mode=040750 ouid=995 ogid=995 rdev=00:00 nametype=PARENT type=PATH msg=audit(1558701993.212:1178): item=1 name="/opt/journalbeat/current/data/" inode=385637 dev=08:01 mode=040750 ouid=995 ogid=995 rdev=00:00 nametype=PARENT type=PATH msg=audit(1558701993.212:1178): item=2 name="/opt/journalbeat/current/data/registry.new" inode=388710 dev=08:01 mode=0100600 ouid=0 ogid=0 rdev=00:00 nametype=DELETE type=PATH msg=audit(1558701993.212:1178): item=3 name="/opt/journalbeat/current/data/registry" inode=384667 dev=08:01 mode=0100600 ouid=995 ogid=995 rdev=00:00 nametype=DELETE type=PATH msg=audit(1558701993.212:1178): item=4 name="/opt/journalbeat/current/data/registry" inode=388710 dev=08:01 mode=0100600 ouid=0 ogid=0 rdev=00:00 nametype=CREATE
This is where
registry.new is getting renamed to
registry.new (tmpfile with new state) is getting created as owned by
root:root and then overwrites original
registry that was owned by
journalbeat:journalbeat. This is being called within a go function (run > persist > flush).
- main journalbeat process is running as journalbeat:journalbeat and has been running for days
- audit log shows journalbeat running as root:root with different PID than main process...this seems to be ephemeral as these audit log PIDs don't show up via ps when I look. Guessing this is related to goroutines...but didn't think they ran as separate process and not sure why they would end up running as root. Could also be something funky with systemd.
Any thoughts/insights on why this is happening? I am guessing I am missing something obvious but it hasn't hit me yet.