Logstash can't run as non-privileged user logstash

Elastic Stack Logstash
1

Ubuntu 12.04
Logstash 1.5.3

Logstash launches, but does not forward log files as user 'logstash'. It can (I know, I know, this is Wrong Of Me) as user root. I am confused.

(I will add that I've spent most of the last year messing around with CentOS and RedHat Machines, so I might be Missing Something with Ubuntu.)

shipper conf file

root@ip-172-20-11-51:/etc/logstash/conf.d# ls -ltrh
total 4.0K
-rw-r--r-- 1 logstash logstash 139 Aug 14 19:23 01logstash_shipper.conf

cat 01logstash_shipper.conf
input { file { path => "/var/log/syslog*"} }

output {
redis {
 host => '172.20.255.224'
 data_type => 'list'
 key => 'logstash:redis'
}
}

receiver conf file

    input {
    redis {
      host => 'localhost'
      data_type => 'list'
      key => 'logstash:redis'
      type => 'redis-input'
    }
    } 

output {file { path => "/var/log/combined/syslog"} }

As user Logstash, stripped from the init script so I can see what is going on.

# su - logstash
logstash@ip-172-20-11-51:~$ /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/01logstash_shipper.conf -l /var/log/logstash/test2.log

Nothing in output to the destination file /var/log/combined/syslog

As user root ...

root@ip-172-20-11-51:/etc/logstash/conf.d# /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/01logstash_shipper.conf -l /var/log/logstash/test2.log
Sending logstash logs to /var/log/logstash/test2.log.

And output just flies into it ...

root@ip-172-20-11-51:/etc/logstash/conf.d# logger TEST7
root@ip-172-20-11-51:/etc/logstash/conf.d#


tail -f /var/log/combined/syslog


{"message":"Aug 17 19:56:40 ip-172-20-11-51 ubuntu: TEST6","@version":"1","@timestamp":"2015-08-17T19:56:46.629Z","host":"0.0.0.0","path":"/var/log/syslog","type":"redis-input"}
2

First of all you need to make sure the logstash user is a member of the adm group so that it has permissions to read e.g. /var/log/syslog.

Secondly, there's a bug in 12.04's Upstart that prevents secondary groups from being added to started processes (i.e. Logstash processes started via Upstart won't actually be a member of e.g. the adm group regardless of what /etc/group says). Logstash implemented a fix for this (github.com/elastic/logstash pull request #1398) that I believe is included in Logstash 1.5.3 so you should be okay.

(Side note: If you really want to use /var/log/syslog* as a wildcard, make sure you also exclude *.gz.)

1 Like
3

Adding the user to group adm did the trick. And thanks, I eliminated the wildcard - that was not what I intended.