Kibana service fails to start after upgrading from 4.5.1 to 5.0.0-alpha4

Elastic Stack Kibana
1

Hi !

I would like to confirm what I am trying to do is right thing or not.
It looks like similar issue to https://github.com/elastic/kibana/issues/7475
but since its upgrade issue , I have created a new thread.

Problem
After upgrading kibana from 4.5.1 to 5.0.0-alpha by yum update , kibana fails to start as below.

Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service holdoff time over, scheduling restart.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: Started Kibana.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: Starting Kibana...
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[12491]: Failed at step USER spawning /usr/share/kibana/bin/kibana: No such process
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service: main process exited, code=exited, status=217/USER
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: Unit kibana.service entered failed state.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service failed.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service holdoff time over, scheduling restart.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: start request repeated too quickly for kibana.service
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: Failed to start Kibana.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: Unit kibana.service entered failed state.
Jul 08 15:17:55 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service failed.

Analysis
After checking the unit file for systemd, kibana now starts up with user kibana.

[root@ip-172-30-1-196 logstash]# cat /usr/lib/systemd/system/kibana.service
[Unit]
Description=Kibana

[Service]
Type=simple
User=kibana
Group=kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always

[Install]
WantedBy=multi-user.target

After creating user kibana , systemd still has errors

[root@ip-172-30-1-196 logstash]# useradd  -M -s /sbin/nologin kibana
[root@ip-172-30-1-196 logstash]# grep kibana /etc/passwd
kibana:x:1001:1001::/home/kibana:/sbin/nologin

Jul 08 15:30:39 ip-172-30-1-196.ec2.internal systemd[1]: kibana.service holdoff time over, scheduling restart.
Jul 08 15:30:39 ip-172-30-1-196.ec2.internal systemd[1]: Started Kibana.
Jul 08 15:30:39 ip-172-30-1-196.ec2.internal systemd[1]: Starting Kibana...
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: fs.js:549
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: ^
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: Error: EACCES: permission denied, open '/usr/share/kibana/optimize/.babelcache.json'
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at Error (native)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at Object.fs.openSync (fs.js:549:18)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at Object.fs.writeFileSync (fs.js:1156:15)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at save (/usr/share/kibana/node_modules/babel-core/lib/api/register/cache.js:35:19)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at nextTickCallbackWith0Args (node.js:420:9)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at process._tickDomainCallback (node.js:390:13)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at Function.Module.runMain (module.js:443:11)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at startup (node.js:139:18)
Jul 08 15:30:40 ip-172-30-1-196.ec2.internal kibana[12623]: at node.js:968:3

To bypass this error , I now start up kibana with user root and group root.

Question
My question is should I chown the whole kibana env as below after yum update?

chown -R kibana:kibana /usr/share/kibana

Thanks,
Yu

2

Can you please raise this on Github?

3

Sure. No problem.

1 Like
4

Git hub link is

1 Like
5

After chowning the "/usr/share/kibana/optimize" kibana starts up properly.

Before chowning

[root@ip-172-30-1-196 ec2-user]# cd /usr/share/kibana/
[root@ip-172-30-1-196 kibana]# ls -ltr
total 20
-rw-rw-r--.  1 root root 2445 Jun 30 02:57 README.txt
-rw-rw-r--.  1 root root  563 Jun 30 02:57 LICENSE.txt
drwxrwxr-x.  2  993  990    6 Jun 30 02:57 installedPlugins
-rw-rw-r--.  1 root root  708 Jun 30 02:57 package.json
drwxr-xr-x. 95 root root 4096 Jul  8 14:33 node_modules
drwxr-xr-x.  9 root root   96 Jul  8 14:33 src
drwxr-xr-x.  2 root root   39 Jul  8 14:33 bin
drwxr-xr-x.  6 root root  101 Jul  8 14:33 node
drwxr-xr-x.  3  993  990   43 Jul  8 14:33 optimize
drwxr-xr-x.  2 root root 4096 Jul  8 14:34 webpackShims

Chown command

[root@ip-172-30-1-196 kibana]# chown kibana:kibana -R /usr/share/kibana/optimize/

My systemd script

[root@ip-172-30-1-196 kibana]# cat /usr/lib/systemd/system/kibana.service
[Unit]
Description=Kibana

[Service]
Type=simple
User=kibana
Group=kibana
#User=root
#Group=root
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always

[Install]
WantedBy=multi-user.target

Make sure kibana user exists

[root@ip-172-30-1-196 kibana]# grep kibana /etc/passwd
kibana:x:1001:1001::/home/kibana:/sbin/nologin

Service startup and result

[root@ip-172-30-1-196 kibana]# systemctl daemon-reload
[root@ip-172-30-1-196 kibana]# systemctl stop kibana
[root@ip-172-30-1-196 kibana]# systemctl start kibana
[root@ip-172-30-1-196 kibana]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/usr/lib/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-07-13 10:06:21 JST; 50s ago
 Main PID: 2278 (node)
   CGroup: /system.slice/kibana.service
           └─2278 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.y...

Jul 13 10:06:23 ip-172-30-1-196.ec2.internal kibana[2278]: {"type":"log","@timestamp":"2016-07-13T01:06:23Z","...d"}
Jul 13 10:06:23 ip-172-30-1-196.ec2.internal kibana[2278]: {"type":"log","@timestamp":"2016-07-13T01:06:23Z","...d"}
Jul 13 10:06:23 ip-172-30-1-196.ec2.internal kibana[2278]: {"type":"log","@timestamp":"2016-07-13T01:06:23Z","...d"}
Jul 13 10:06:23 ip-172-30-1-196.ec2.internal kibana[2278]: {"type":"log","@timestamp":"2016-07-13T01:06:23Z","...d"}
Jul 13 10:06:23 ip-172-30-1-196.ec2.internal kibana[2278]: {"type":"log","@timestamp":"2016-07-13T01:06:23Z","...h"}
Jul 13 10:06:23 ip-172-30-1-196.ec2.internal kibana[2278]: {"type":"log","@timestamp":"2016-07-13T01:06:23Z","...d"}
Jul 13 10:06:23 ip-172-30-1-196.ec2.internal kibana[2278]: {"type":"log","@timestamp":"2016-07-13T01:06:23Z","...d"}
Jul 13 10:06:23 ip-172-30-1-196.ec2.internal kibana[2278]: {"type":"log","@timestamp":"2016-07-13T01:06:23Z","...1"}
Jul 13 10:06:23 ip-172-30-1-196.ec2.internal kibana[2278]: {"type":"log","@timestamp":"2016-07-13T01:06:23Z","...d"}
Jul 13 10:06:35 ip-172-30-1-196.ec2.internal systemd[1]: Started Kibana.
Hint: Some lines were ellipsized, use -l to show in full.

Thanks,
Yu