Kibana already started but the status is not running

fadzhli · February 25, 2016, 4:32am

I'm just updated my Kibana to 4.4 and I have encounter some problems:

[fadzhli@elk ~]$ sudo service kibana start
chown: invalid user: 'kibana:root'
kibana started

[fadzhli@elk ~]$ sudo service kibana status
[sudo] password for fadzhli: 
kibana is not running

As you can see I start the service, the service is running but on the status is not running. How to resolve this problem,

Thank you

LeeDr · February 25, 2016, 6:08pm

The first message "kibana started" just means it was started, but apparently failed and now is not running.

You probably need to look in /var/log/kibana. There should be a stdout and stderr file. If that doesn't tell you the problem please paste it here and someone will try to help.

Regards,
Lee

carlosq · February 25, 2016, 6:27pm

I have same problem and I checked log file /var/log/kibana/kibana.stderr, it has errors as:

Error: Cannot find module '/opt/kibana/src/src/utils/packageJson'
at Function.Module._resolveFilename (module.js:336:15)
at Function.Module._load (module.js:278:25)
at Module.require (module.js:365:17)
at require (module.js:384:17)
at requireModule (/opt/kibana/src/node_modules/requirefrom/index.js:24:10)
at Object. (/opt/kibana/src/cli/cli.js:6:11)
at Module._compile (module.js:460:26)
at Module._extensions..js (module.js:478:10)
at Object.require.extensions.(anonymous function) [as .js] (/opt/kibana/node_modules/babel-core/lib/api/register/node.js:214:7)
at Module.load (module.js:355:32)

fadzhli · February 26, 2016, 4:33am

Hi, although the status is not running, my kibana works just fine and this is weird.

My stderr output is

chroot: invalid user
chroot: invalid user
chroot: invalid user
chroot: invalid user
chroot: invalid user
chroot: invalid user
chroot: invalid user
chroot: invalid user
chroot: invalid user
chroot: invalid user

My stdout is

{"type":"log","@timestamp":"2016-02-22T09:26:32+00:00","tags":["error","elasticsearch"],"pid":1436,"message":"Request error, retrying -- connect ECONNREFUSED"}
{"type":"log","@timestamp":"2016-02-22T09:26:32+00:00","tags":["status","plugin:markdown_vis","info"],"pid":1436,"name":"plugin:markdown_vis","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-02-22T09:26:32+00:00","tags":["status","plugin:metric_vis","info"],"pid":1436,"name":"plugin:metric_vis","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-02-22T09:26:32+00:00","tags":["warning","elasticsearch"],"pid":1436,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2016-02-22T09:26:32+00:00","tags":["warning","elasticsearch"],"pid":1436,"message":"No living connections"}
{"type":"log","@timestamp":"2016-02-22T09:26:32+00:00","tags":["status","plugin:elasticsearch","error"],"pid":1436,"name":"plugin:elasticsearch","state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://localhost:9200. Retrying in 2.5 seconds.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2016-02-22T09:26:32+00:00","tags":["status","plugin:spyModes","info"],"pid":1436,"name":"plugin:spyModes","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-02-22T09:26:32+00:00","tags":["status","plugin:statusPage","info"],"pid":1436,"name":"plugin:statusPage","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-02-22T09:26:32+00:00","tags":["status","plugin:table_vis","info"],"pid":1436,"name":"plugin:table_vis","state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2016-02-22T09:26:32+00:00","tags":["fatal"],"pid":1436,"level":"fatal","message":"listen EADDRINUSE","error":{"message":"listen EADDRINUSE","name":"Error","stack":"Error: listen EADDRINUSE\n    at exports._errnoException (util.js:746:11)\n    at Server._listen2 (net.js:1156:14)\n    at listen (net.js:1182:10)\n    at net.js:1280:9\n    at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:81:16)\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:99:10)","code":"EADDRINUSE"}}

Thank you

LeeDr · February 26, 2016, 5:17pm

Hi Mohd,

Let's go back and try to figure out why you're getting this invalid user error;

Can you check this;
ls -l /opt/kibana

On my install I see the optimize folder is owned by kibana in the root group;
drwxrwxr-x 3 kibana root 4096 Feb 25 17:58 optimize

Does you're look the same?

Or another thing to check is
leedr@U14K:~$ grep kibana /etc/passwd kibana:x:998:998::/home/kibana:

If you don't have the kibana user we need to see how you installed kibana and maybe go through that process again.

BY THE WAY, there is a 4.4.1 kibana release now that has a security fix.

Also, the last stdout message is a fatal message saying that the address is in use. So you may have 2 instances of kibana trying to start. You can check with ps -ef | grep kibana

fadzhli · March 1, 2016, 7:37am

Hi Lee,

Thank you for your response.

drwxr-xr-x 3 990 989 43 Jan 19 11:16 optimize

I don't have kibana user on my /etc/passwd & I installed my kibana by doing ** yum update kibana**

Can you show me how to add user kibana back on the server.

Thank you

LeeDr · March 1, 2016, 2:47pm

Hi Mohd,

I'm not sure which version of Kibana you had installed prior to your "yum update", but there may have been some changes to the packaging in the way the kibana user is created. And a problem has recently been reported that the upgrade process tries to delete and then recreate the kibana user which it should not do.

And yes, there is a 4.4.1 release now with a security fix.

The easiest way to get back on track would be to uninstall Kibana, and then install Kibana 4.4.1.

If you have any modifications to your kibana/config/kibana.yml save a copy of that in a different directory.

Regards,
Lee

fadzhli · March 2, 2016, 3:43am

Hi Lee

I've already uninstall Kibana and install Kibana 4.4.1.

sh-4.2$ sudo service kibana start
kibana started
sh-4.2$ sudo service kibana status
kibana is not running

Kibana user also in /etc/passwd

Funny thing is, the kibana service is running perfectly but when I check the status is not running.

Then I check ls -l /opt/kibana and my kibana is not in root group.

So I run

sudo chown -R kibana:kibana /opt/kibana

Now the output of my kibana status is ok

sh-4.2$ sudo service kibana start
kibana started
sh-4.2$ sudo service kibana status
kibana is running

But I have this error on stderr

Error: EACCES, permission denied '/opt/kibana/optimize/.babelcache.json'
    at Error (native)
    at Object.fs.openSync (fs.js:500:18)
    at Object.fs.writeFileSync (fs.js:1099:15)
    at save (/opt/kibana/node_modules/babel-core/lib/api/register/cache.js:35:19)
    at process._tickDomainCallback (node.js:381:11)
    at Function.Module.runMain (module.js:503:11)
    at startup (node.js:129:16)
    at node.js:814:3

LeeDr · March 2, 2016, 2:27pm

Hi Mohd,

Here's the permissions on that .babelcache.json on my system where I just did a clean install and service start on kibana;

root@U14K:~# ls -al /opt/kibana/optimize/
total 16
drwxrwxr-x  3 kibana root 4096 Mar  2 08:14 .
drwxrwxr-x 10 root   root 4096 Mar  2 08:12 ..
-rw-r--r--  1 kibana root    2 Mar  2 08:14 .babelcache.json
drwxrwxr-x  4 root   root 4096 Mar  2 08:12 bundles

I'm guessing your .babelcache.json is owned by root instead of kibana? If that's true, I would stop your kibana service and change the owner to kibana:root like this one. Then start kibana and see that you don't get that error any longer.

Since the kibana service runs as user kibana, it's probably OK to do a recursive chown on the entire /opt/kibana directory to kibana:kibana.

Regards,
Lee

fadzhli · March 3, 2016, 9:20am

Hi Lee,

I've change the owner to kibana:root

sh-4.2$ ls -al /opt/kibana/optimize/
total 12
drwxr-xr-x  3 kibana root   43 Jan 19 11:16 .
drwxr-xr-x 10 kibana root 4096 Mar  2 10:26 ..
-rw-r--r--  1 kibana root    2 Mar  3 17:17 .babelcache.json
drwxr-xr-x  4 kibana root 4096 Mar  2 10:26 bundles

But now I have this new error in my stderr log

FATAL { [Error: listen EADDRINUSE]
  cause: { [Error: listen EADDRINUSE] code: 'EADDRINUSE', errno: 'EADDRINUSE', syscall: 'listen' },
  isOperational: true,
  code: 'EADDRINUSE',
  errno: 'EADDRINUSE',
  syscall: 'listen' }

How to resolve this error?

LeeDr · March 3, 2016, 3:04pm

You must have 2 instances trying to start now. First try service kibana stop
Wait a few seconds for kibana to stop.
Then check with ps -ef | grep kibana
If there's still a kibana process running, you could wait another few seconds and check again, or kill it with kill -9 <pid-of-kibana-process>
Then start it normally service kibana start

fadzhli · March 4, 2016, 9:17am

Hi Lee,

I have kill my kibana process but it seem the process is still there. Although I have error in my stderr log but my kibana work just fine. I think maybe I just can reinstall all the ELK stack later. Or do you have other opinion?

Thank you.

LeeDr · March 14, 2016, 7:15pm

Hi Mohd,

I was out on vacation last week. Do you have Kibana working now? Still have errors in your log?

Lee

fadzhli · March 18, 2016, 3:32am

Hi lee,

My Kibana is working fine. But I still have the error on my stderr

  isOperational: true,
  code: 'EADDRINUSE',
  errno: 'EADDRINUSE',
  syscall: 'listen' }
FATAL { [Error: listen EADDRINUSE]
  cause: { [Error: listen EADDRINUSE] code: 'EADDRINUSE', errno: 'EADDRINUSE', syscall: 'listen' },
  isOperational: true,
  code: 'EADDRINUSE',
  errno: 'EADDRINUSE',
  syscall: 'listen' }

I just ignore it.

Thank you for replying =)

Deepak_Patnaik · September 24, 2016, 5:19pm

Hi,
I am also getting the same error message. Now when I try to remove the Kibana. it gives me the below message.

Package kibana is not installed, so not removed

Need urgent help as I have got a demo scheduled for next week.

Thanks,
Deepak

Deepak_Patnaik · September 24, 2016, 5:22pm

I checked for any running jobs.

deepak 18060 18004 0 22:51 pts/1 00:00:00 grep --color=auto kibana

I tried to kill 18060, but it shows, there is no such job.

LeeDr · September 26, 2016, 12:54pm

Hi Deepak,

When you run something like ps -ef | grep Kibana, the ps -ef includes the process doing the grep and that's the output you see ... grep --color=auto kinana. So you have to ignore that line, or remove it like this;
ps - ef | grep kibana | grep -v grep

For your Kibana issue, can you reply with your operating system, Kibana version, and how you installed it?

Thanks,
Lee

Spandana · October 21, 2016, 2:07pm

Hi Lee,

I have a similar problem. When i try to start kibana
I get this
bash-4.1$ sudo service kibana start
chown: invalid group: `kibana:kibana'
kibana started

and

bash-4.1$ ls -al /opt/kibana/optimize/
total 16
drwxr-xr-x 3 kibana root 4096 Oct 21 05:39 .
drwxr-xr-x 10 kibana root 4096 Oct 21 05:39 ..
-rw-rw-r-- 1 kibana root 2 Oct 21 06:45 .babelcache.json
drwxr-xr-x 4 kibana root 4096 Oct 21 05:39 bundles

Any suggestions on this please. I tried yum update kibana it says 'No Packages marked for Update'

fadzhli · October 21, 2016, 2:35pm

Hi,

What version of kibana you using now?

Spandana · October 22, 2016, 12:55pm

Hi fadzhil,
Its kibana 4.5 version