Run Logstash as service on RedHat

Elastic Stack Logstash
1

Hi,

I have unpack (as application user that will run it) logstash 7.16 to my applications folder on RedHat server.

I made changes in logstash.conf file and to try I started logstash in command line like this:

/applications/logstash-7-16-2/logstash-7.16.2/bin/logstash  -f  /applications/logstash-7-16-2/logstash-7.16.2/config/logstash.conf

It started correctly and I was able to send data from filebeat to elastic.

Then I wanted to start Logstash as service so I created
logstash716.service file under /etc/systemd/system

[Unit]
Description=logstash716

[Service]
Type=simple
User=application_user
Group=application_user

# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.

EnvironmentFile=-/applications/logstash-7-16-2/logstash-7.16.2/
ExecStart=/applications/logstash-7-16-2/logstash-7.16.2/bin/logstash  -f  /applications/logstash-7-16-2/logstash-7.16.2/config/logstash.conf
WorkingDirectory=/applications/logstash-7-16-2/logstash-7.16.2/

Restart=always
Nice=19
LimitNOFILE=16384

# When stopping, how long to wait before giving up and sending SIGKILL?
# Keep in mind that SIGKILL on a process can cause data loss.
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target

Then run:

sudo systemctl daemon-reload
sudo systemctl enable logstash716.service
systemctl start logstash716.service

Unfortunately it failed to start with error Permission denied. In var/log/messages there is:

logstash716.service: Failed to execute command: Permission denied
logstash716.service: Failed at step EXEC spawning /applications/logstash-7-16-2/logstash-7.16.2/bin/logstash: Permission denied
logstash716.service: Service RestartSec=100ms expired, scheduling restart.
logstash716.service: Scheduled restart job, restart counter is at 5.
Stopped logstash716.
logstash716.service: Start request repeated too quickly.
logstash716.service: Failed with result 'exit-code'.
Failed to start logstash716.

I can't find what is wrong with the permissions...?

image
image583×134 42.5 KB

image
image622×266 70.2 KB

image
image578×327 87.9 KB

Any help would be appreciated.

2

Not sure it will help, but the service file for latest (9.x) rpm release looks like this:

# cat /usr/lib/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

# When stopping, how long to wait before giving up and sending SIGKILL?
# Keep in mind that SIGKILL on a process can cause data loss.
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target

Note your EnvironmentFile points at a directory, not a file, but ... should not matter/give error you saw.

The (bash) script that runs /usr/share/logstash/bin/logstash is actually owned by root by the default RHEL installation:

# ls -l /usr/share/logstash/bin/logstash
-rwxr-xr-x. 1 root root 2149 Jun 17 14:05 /usr/share/logstash/bin/logstash

In fact, even when the service is running, and the process is running as logstash user/group, very little has permissions of the logstash user/group:

# find / -xdev \( -group logstash -o -user logstash \) -ls
   471255      0 drwxr-xr-x   2 logstash logstash       18 Jul 15 11:08 /tmp/hsperfdata_logstash
   586925     32 -rw-------   1 logstash logstash    32768 Jul 15 11:09 /tmp/hsperfdata_logstash/2126
 27249083      0 drwxr-xr-x   4 logstash logstash       69 Jul 14 08:37 /var/lib/logstash
 26632194      0 drwxr-xr-x   2 logstash logstash        6 Jul 14 08:37 /var/lib/logstash/queue
  1601397      0 drwxr-xr-x   2 logstash logstash        6 Jul 14 08:37 /var/lib/logstash/dead_letter_queue
 26632205      0 -rw-r--r--   1 logstash logstash        0 Jul 14 08:37 /var/lib/logstash/.lock
 26632207      4 -rw-r--r--   1 logstash logstash       36 Jul 14 08:37 /var/lib/logstash/uuid
  2120777      4 drwxr-xr-x   2 logstash root         4096 Jul 15 11:09 /var/log/logstash
  2120782      0 -rw-r--r--   1 logstash logstash        0 Jul 14 08:37 /var/log/logstash/logstash-json.log
  2120783      0 -rw-r--r--   1 logstash logstash        0 Jul 14 08:37 /var/log/logstash/logstash-slowlog-plain.log
  2120784      0 -rw-r--r--   1 logstash logstash        0 Jul 14 08:37 /var/log/logstash/logstash-deprecation.log
  2120785      0 -rw-r--r--   1 logstash logstash        0 Jul 14 08:37 /var/log/logstash/logstash-slowlog-json.log
  2138807  14292 -rw-r--r--   1 logstash logstash 14631735 Jul 15 11:09 /var/log/logstash/logstash-plain-2025-07-14-1.log.gz
  2138808      8 -rw-r--r--   1 logstash logstash     5417 Jul 15 11:09 /var/log/logstash/logstash-plain.log

# ps -ww -ulogstash -ouser,euser,group,egroup,command
USER     EUSER    GROUP    EGROUP   COMMAND
logstash logstash logstash logstash /usr/share/logstash/jdk/bin/java
3

check SElinux status (via getenforce) and ausearch -m avc -ts recent output if it is set to "Enforcing", and setenforce 0 to reset it, and try again.

4

getenforce returns Enforcing
I am not sure if I can change it. I need to consult with Linux admin.

"EnvironmentFile=-/applications/logstash-7-16-2/logstash-7.16.2/" as I understand it doesn't matter when "-" is at beginning.

5

Good luck.

You can address this (at least) 2 ways. 1. turn off Enforcing. 2. add your directories to the paths SElinux allows. The latter is probably a "better" fix. But if you (Joe Normal User) are allowed to do that, then "Enforcing" would have limited value.

Yes, but the broken setting adds zero value. "Here's an error, but please ignore the error". You might as well point it at /dev/null.

6

setenforce 0 did the trick but it cannot stay that way. Looks like I need to look into permissions deeper.

@RainTown thank you for your time!

1 Like