Kibana is not running

Elastic Stack Kibana
1

Installed ELK on AWS ubuntu 14.04. ELK versions : Elasticsearch 2.2.x, Logstash 2.2.x, and Kibana 4.4.x. While checking the status elasticsearch and logstash are running, but kibana is not running.
I will share the result while running kibana...

Last lines of sudo /opt/kibana/bin/kibana
log [09:51:21.184] [info][listening] Server running at http://localhost:5601
log [09:51:51.119] [error][status][plugin:elasticsearch] Status changed from yellow to red - Request Timeout after 30000ms

./bin/kibana

fs.js:565
fs.write = function(fd, buffer, offset, length, position, callback) {
^
Error: EACCES, permission denied '/opt/kibana/optimize/.babelcache.json'
at Error (native)
at Object.fs.openSync (fs.js:500:18)
at Object.fs.writeFileSync (fs.js:1099:15)
at save (/opt/kibana/node_modules/babel-core/lib/api/register/cache.js:35:19)
at process._tickDomainCallback (node.js:381:11)
at Function.Module.runMain (module.js:503:11)
at startup (node.js:129:16)
at node.js:814:3

Log information

tail kibana.stdout gives the information

{"type":"response","@timestamp":"2016-02-24T11:07:14+00:00","tags":[],"pid":1123,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/logstash-/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"52.72.247.248","content-length":"178","accept":"application/json, text/plain, /","origin":"http://52.72.247.248","kbn-version":"4.4.1","user-agent":"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://52.72.247.248/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"172.30.0.28","userAgent":"172.30.0.28","referer":"http://52.72.247.248/app/kibana?"},"res":{"statusCode":200,"responseTime":109,"contentLength":9},"message":"POST /elasticsearch/logstash-/_field_stats?level=indices 200 109ms - 9.0B"}
{"type":"response","@timestamp":"2016-02-24T11:07:15+00:00","tags":[],"pid":1123,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1456301293544","method":"post","headers":{"connection":

When I start kibana, it will be running for sometime. But within seconds status will change to not running. I don't understand what is happening behind. Previously logserver was working perfectly, but once I stop and start the aws instance this issue came.
Please help me to solve.

Thanks

2

What are the permissions on the file /opt/kibana/optimize/.babelcache.json?

run:

ls -l /opt/kibana/optimize/.babelcache.json

and paste the output.

Also, why are you running kibana with sudo? Running it with root privileges is probably not ideal.

3

-rwxr-xr-x 1 kibana kibana 2 Feb 26 10:21 /opt/kibana/optimize/.babelcache.json

Without sudo I got the message I put under ./bin/kibana
So I tried with sudo. Also with sudo , the server status page will load, and it will show as waiting for plugin elasticsearch, and after few seconds "Request Timout " message will come.

4

I am here sharing elasticsearch log also,,,

at
org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:245)
at
org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-02-25 11:19:32,106][WARN ][index.engine ] [ontash_prod] [logstash-2016.02.24][2] failed engine [create]
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImubuntu

5

That probably explains it...

java.io.IOException: No space left on device

Do you have a full disk?

6

If you can access the Elasticsearch HTTP API you can delete old indexes with a curl command. Something like:

curl -XDELETE http://localhost:9200/logstash-2016.01.*

will remove all logs from January (2016.01).

Take a look at Curator for managing indexes (deleting or moving old data).

7

I think space issue is the problem, I chkecked instance system information, it gives
System load: 0.0 Memory usage: 2% Processes: 80
Usage of /: 96.2% of 29.39GB Swap usage: 0% Users logged in: 0
So I will try deleting logs as you said. And will let you know the result.

8

I uninstalled elasticsearch. will start from begining. But still system is using 76% of memory.
Is there any hidden folders or something which keep old logs??

9

DId you manage to get to the bottom of this? I'm not sure of your configuration (volumes / data storage configuration).

10

Hi,

I removed elasticsearch completely and reinstalled it. Configured elasticsearch and kibana correctly. It works .
I am trying to move logs to aws s3 to save disk space in ec2.

Thanks