Fail to start kibana, the kibana.pid file not accessable

Elastic Stack Kibana
1

I installed kibana in AWS EC2 instance. It has problem to start.

The kibana.pid is in /var/run/. I have changed the mode of the directory to 777.

ll -d /var/run

lrwxrwxrwx 1 root root 4 Jun 19 23:49 /var/run -> /run/

ll -d /run

drwxrwxrwx 31 root root 1160 Jul 13 10:03 /run/

But when I start it gave the log below, just could not access the pid file.

{"type":"log","@timestamp":"2017-07-13T09:14:23Z","tags":["fatal"],"pid":24354,"level":"fatal","message":"EACCES: permission denied, open '/var/run/kibana.pid'","error":{"message":"EACCES: permission denied, open '/var/run/kibana.pid'","name":"Error","stack":"Error: EACCES: permission denied, open '/var/run/kibana.pid'\n at Error (native)","code":"EACCES"}}

I even tried to change the pid file location, but it performed as the same.

If change the owner of the kibana.pid, it failed to start either.

{"type":"log","@timestamp":"2017-07-13T09:53:37Z","tags":["fatal"],"pid":31969,"level":"fatal","message":"listen EADDRNOTAVAIL 52.89.77.102:5601","error":{"message":"listen EADDRNOTAVAIL 52.89.77.102:5601","name":"Error","stack":"Error: listen EADDRNOTAVAIL 52.89.77.102:5601\n at Object.exports._errnoException (util.js:1018:11)\n at exports._exceptionWithHostPort (util.js:1041:20)\n at Server._listen2 (net.js:1249:19)\n at listen (net.js:1298:10)\n at net.js:1408:9\n at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:62:16)\n at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:81:10)","code":"EADDRNOTAVAIL"}}
FATAL CLI ERROR Error: ENOENT: no such file or directory, unlink '/var/run/kibana.pid'
at Error (native)
at fs.unlinkSync (fs.js:1103:18)
at process. (/usr/share/kibana/src/server/pid/index.js:52:26)
at process. (/usr/share/kibana/node_modules/lodash/index.js:7363:25)
at process.g (events.js:291:16)
at emitOne (events.js:101:20)
at process.emit (events.js:188:7)
at process.exit (internal/process.js:164:15)
at Command. (/usr/share/kibana/src/cli/serve/serve.js:138:17)
at throw (native)
at step (/usr/share/kibana/src/cli/serve/serve.js:25:191)
at /usr/share/kibana/src/cli/serve/serve.js:25:402

Can someone help please.

2

How did you originally install Kibana? What command are you using to start Kibana? Are you running Kibana as the kibana user?

3

I used "/etc/init.d/kibana start" as root to start it.

-rw-r--r-- 1 root root 6 Jul 14 02:35 /var/run/kibana.pid

The elasticsearch was started by elasticsearch user.

-rw-r--r-- 1 elasticsearch elasticsearch 5 Jul 13 09:58 elasticsearch.pid

kibana:x:114:119::/home/kibana:/bin/false
elasticsearch:x:115:120::/home/elasticsearch:/bin/false

I followed the step by step to start kibana, there should be a reason it failed.

4

It shouldn't be necessary to run kibana as root, you should be able to run it as the kibana user. What happens if you do so? Did you install via deb, RPM, or tar?

Also which version are you running?

5

I had an issue when I upgraded Kibana. It wanted to write the pid in /var/run/kibana/kibana.pid but the kibana directory under /var/run didn't exist. I created the directory and chown kibana:kibana and then it started fine.

Also, even if you start kibana as root inside the init script it might change who the service is running as.

You might try deleting by hand the existing pid file.

1 Like
6

I installed it via tar.
The kibana user that is "/bin/false" cannot "su" it.

It continuly restart.

systemctl status kibana.service
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2017-07-17 02:16:08 UTC; 84ms ago
Process: 25475 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=1/FAILURE)
Main PID: 25475 (code=exited, status=1/FAILURE)

Jul 17 02:16:08 ip-172-31-6-106 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Jul 17 02:16:08 ip-172-31-6-106 systemd[1]: kibana.service: Unit entered failed state.
Jul 17 02:16:08 ip-172-31-6-106 systemd[1]: kibana.service: Failed with result 'exit-code'.

7

I chown it kibana:kibana, at the beginning it runs well, but after a while it stopped.

8

I changed the mode of kibana.pid file to 777, and the owner of it to kibana:kibana.
But still not work.

{"type":"log","@timestamp":"2017-07-13T09:53:37Z","tags":["fatal"],"pid":31969,"level":"fatal","message":"listen EADDRNOTAVAIL 52.89.77.102:5601","error":{"message":"listen EADDRNOTAVAIL 52.89.77.102:5601","name":"Error","stack":"Error: listen EADDRNOTAVAIL 52.89.77.102:5601\n at Object.exports._errnoException (util.js:1018:11)\n at exports._exceptionWithHostPort (util.js:1041:20)\n at Server._listen2 (net.js:1249:19)\n at listen (net.js:1298:10)\n at net.js:1408:9\n at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:62:16)\n at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:81:10)","code":"EADDRNOTAVAIL"}}
FATAL CLI ERROR Error: ENOENT: no such file or directory, unlink '/var/run/kibana.pid'
at Error (native)
at fs.unlinkSync (fs.js:1103:18)
at process. (/usr/share/kibana/src/server/pid/index.js:52:26)
at process. (/usr/share/kibana/node_modules/lodash/index.js:7363:25)
at process.g (events.js:291:16)
at emitOne (events.js:101:20)
at process.emit (events.js:188:7)
at process.exit (internal/process.js:164:15)
at Command. (/usr/share/kibana/src/cli/serve/serve.js:138:17)
at throw (native)
at step (/usr/share/kibana/src/cli/serve/serve.js:25:191)
at /usr/share/kibana/src/cli/serve/serve.js:25:402
{"type":"log","@timestamp":"2017-07-14T02:35:51Z","tags":["pid","warning"],"pid":"29971","path":"/var/run/kibana.pid","message":"pid file already exists at /var/run/kibana.pid"}
{"type":"log","@timestamp":"2017-07-14T02:35:51Z","tags":["fatal"],"pid":29971,"level":"fatal","message":"EACCES: permission denied, open '/var/run/kibana.pid'","error":{"message":"EACCES: permission denied, open '/var/run/kibana.pid'","name":"Error","stack":"Error: EACCES: permission denied, open '/var/run/kibana.pid'\n at Error (native)","code":"EACCES"}}
{"type":"log","@timestamp":"2017-07-17T02:00:39Z","tags":["pid","warning"],"pid":"21276","path":"/var/run/kibana.pid","message":"pid file already exists at /var/run/kibana.pid"}
{"type":"log","@timestamp":"2017-07-17T02:00:39Z","tags":["fatal"],"pid":21276,"level":"fatal","message":"EACCES: permission denied, open '/var/run/kibana.pid'","error":{"message":"EACCES: permission denied, open '/var/run/kibana.pid'","name":"Error","stack":"Error: EACCES: permission denied, open '/var/run/kibana.pid'\n at Error (native)","code":"EACCES"}}
{"type":"log","@timestamp":"2017-07-17T03:07:48Z","tags":["pid","warning"],"pid":"6863","path":"/var/run/kibana.pid","message":"pid file already exists at /var/run/kibana.pid"}
{"type":"log","@timestamp":"2017-07-17T03:07:48Z","tags":["fatal"],"pid":6863,"level":"fatal","message":"EACCES: permission denied, open '/var/run/kibana.pid'","error":{"message":"EACCES: permission denied, open '/var/run/kibana.pid'","name":"Error","stack":"Error: EACCES: permission denied, open '/var/run/kibana.pid'\n at Error (native)","code":"EACCES"}}

9

Based on the most recent error you posted, it looks like there's already a process listening to the port you're trying to bind to. Perhaps there's still a Kibana process running.

Try lsof -i :5601 to see what's listening on that port and kill the process if necessary.

11

lsof -i :5601 returns nothing.
It just retured "permission denied, open '/var/run/kibana.pid'" every time when I run command "/etc/init.d/kibana start".

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.