Journalbeat creates registry file with root:root ownership instead of journalbeat user

matp · May 24, 2019, 12:57pm

journalbeat 6.7.1 tarball
journalbeat is run via systemd unit with user/group journalbeat/journalbeat
Ubuntu 16.04

# ps -C journalbeat o lstart,pid,user:14,group:14,cmd
                 STARTED   PID USER           GROUP          CMD
Tue May 21 13:33:36 2019  1524 journalbeat    journalbeat    /opt/journalbeat/current/journalbeat run --path.config /opt/journalbeat/ --c journalbeat.yml

Everything is working fine.

I noticed that our provisioning/config-management was "fixing" file ownership in the journalbeat install every few minutes. This turns out to be the journalbeat registry file which is somehow getting ownership set to root:root.

changed ownership of '/opt/journalbeat/journalbeat-6-7-1/data/registry' from root:root to journalbeat:journalbeat

I used auditd to confirm journalbeat was making the change. You can see below that journalbeat is deleting the old registry file and creating a new one with uid/gid 0 (root).

You can also see in the SYSCALL line that the uid/gid of the journalbeat process is 0 (root)...which I don't really understand since it should be running as 995 (journalbeat).

Also the PID in the audit log SYSCALL line doesn't match the PID from ps...each registry relevant SYSCALL in the audit (I only show one) has a different PID. Not sure if that is relevant.

# auditctl -w /opt/journalbeat/journalbeat-6-7-1/data/registry -p wax
# grep journalbeat /var/log/audit/audit.log | grep -v CONFIG_CHANGE
type=SYSCALL msg=audit(1558701993.212:1178): arch=c000003e syscall=264 success=yes exit=0 a0=ffffffffffffff9c a1=c4200446f0 a2=ffffffffffffff9c a3=c420044720 items=5 ppid=9618 pid=8121 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="journalbeat" exe="/opt/journalbeat/journalbeat-6-7-1/journalbeat" key=(null)
type=CWD msg=audit(1558701993.212:1178):  cwd="/opt/journalbeat/journalbeat-6-7-1"
type=PATH msg=audit(1558701993.212:1178): item=0 name="/opt/journalbeat/current/data/" inode=385637 dev=08:01 mode=040750 ouid=995 ogid=995 rdev=00:00 nametype=PARENT
type=PATH msg=audit(1558701993.212:1178): item=1 name="/opt/journalbeat/current/data/" inode=385637 dev=08:01 mode=040750 ouid=995 ogid=995 rdev=00:00 nametype=PARENT
type=PATH msg=audit(1558701993.212:1178): item=2 name="/opt/journalbeat/current/data/registry.new" inode=388710 dev=08:01 mode=0100600 ouid=0 ogid=0 rdev=00:00 nametype=DELETE
type=PATH msg=audit(1558701993.212:1178): item=3 name="/opt/journalbeat/current/data/registry" inode=384667 dev=08:01 mode=0100600 ouid=995 ogid=995 rdev=00:00 nametype=DELETE
type=PATH msg=audit(1558701993.212:1178): item=4 name="/opt/journalbeat/current/data/registry" inode=388710 dev=08:01 mode=0100600 ouid=0 ogid=0 rdev=00:00 nametype=CREATE

This is where registry.new is getting renamed to registry:

github.com

elastic/beats/blob/v6.7.1/journalbeat/checkpoint/checkpoint.go#L319


	}


	_, err = file.Write(data)
	if err != nil {
		file.Close()
		return fmt.Errorf("Failed to flush state to disk. Could not write to "+
			"%s. %v", tempFile, err)
	}


	file.Close()
	err = os.Rename(tempFile, c.file)
	return err
}


// read loads the persisted state from disk. If the file does not exists then
// the method returns nil and no error.
func (c *Checkpoint) read() (*PersistedState, error) {
	c.fileLock.RLock()
	defer c.fileLock.RUnlock()


	contents, err := ioutil.ReadFile(c.file)

So registry.new (tmpfile with new state) is getting created as owned by root:root and then overwrites original registry that was owned by journalbeat:journalbeat. This is being called within a go function (run > persist > flush).

Summing up:

main journalbeat process is running as journalbeat:journalbeat and has been running for days
audit log shows journalbeat running as root:root with different PID than main process...this seems to be ephemeral as these audit log PIDs don't show up via ps when I look. Guessing this is related to goroutines...but didn't think they ran as separate process and not sure why they would end up running as root. Could also be something funky with systemd.

Any thoughts/insights on why this is happening? I am guessing I am missing something obvious but it hasn't hit me yet.

matp · August 20, 2019, 1:00pm

Just a note that this behaviour is still present in 7.3.0.

It doesn't really break anything, just creates noise as we try to enforce "correct" ownership and perms.

Perhaps there is a good reason this is registry file is being re-created with root:root ownership, would be great to understand if this is by design (expected behaviour).

# chown --changes -R journalbeat:journalbeat /opt/journalbeat
changed ownership of '/opt/journalbeat/journalbeat-7-3-0/data/registry' from root:root to journalbeat:journalbeat