Kibana sporadically crashing and not automatically recovering even though the correct systemd UNIT file is in place

[thomas.heuberger]

So, from time to time Kibana sporadically crashes and the service is not automatically restarted, even though the correct systemd UNIT file is in place.

:~$ cat kibana.service
[Unit]
Description=Kibana
StartLimitIntervalSec=30
StartLimitBurst=3

[Service]
Type=simple
User=kibana
Group=kibana
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always
WorkingDirectory=/

[Install]
WantedBy=multi-user.target
root@inpunpr13:/etc/systemd/system#

yet:

:~$ systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Sat 2019-03-16 16:37:19 CET; 1 day 15h ago
 Main PID: 39453 (code=exited, status=0/SUCCESS)

and

:~$ systemctl show kibana.service -p NRestarts
NRestarts=0

Any idea how to prevent / mitigate this? I know this is rather a systemd question than a Kibana question, but I thought that probably somebody encountered this problem as well.

Thanks!

[tiagocosta]

@thomas.heuberger can you try to add RestartSec=5 under the [Service] section of your kibana.service?

[thomas.heuberger]

I modified kibana.service to

:~# cat /etc/systemd/system/kibana.service
[Unit]
Description=Kibana
StartLimitIntervalSec=30
StartLimitBurst=5

[Service]
Type=simple
User=kibana
Group=kibana
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always
RestartSec=30
WorkingDirectory=/

[Install]
WantedBy=multi-user.target

and that seemes to do the trick. I'm not sure why it didn't work without explicitly setting
RestartSec

[tiagocosta]

@thomas.heuberger awesome! Maybe we can mark the answer as the solution until we permanently fix that?

Also it would be very cool to know what is the kibana version you are running right now

[thomas.heuberger]

:~$ /usr/share/kibana/bin/kibana --version
6.6.2