Kibana is not running

(Babeesh) #1

Installed ELK on AWS ubuntu 14.04. ELK versions : Elasticsearch 2.2.x, Logstash 2.2.x, and Kibana 4.4.x. While checking the status elasticsearch and logstash are running, but kibana is not running.
I will share the result while running kibana...

Last lines of sudo /opt/kibana/bin/kibana
log [09:51:21.184] [info][listening] Server running at http://localhost:5601
log [09:51:51.119] [error][status][plugin:elasticsearch] Status changed from yellow to red - Request Timeout after 30000ms


fs.write = function(fd, buffer, offset, length, position, callback) {
Error: EACCES, permission denied '/opt/kibana/optimize/.babelcache.json'
at Error (native)
at Object.fs.openSync (fs.js:500:18)
at Object.fs.writeFileSync (fs.js:1099:15)
at save (/opt/kibana/node_modules/babel-core/lib/api/register/cache.js:35:19)
at process._tickDomainCallback (node.js:381:11)
at Function.Module.runMain (module.js:503:11)
at startup (node.js:129:16)
at node.js:814:3

Log information

tail kibana.stdout gives the information

{"type":"response","@timestamp":"2016-02-24T11:07:14+00:00","tags":[],"pid":1123,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/logstash-/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"","content-length":"178","accept":"application/json, text/plain, /","origin":"","kbn-version":"4.4.1","user-agent":"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"","userAgent":"","referer":""},"res":{"statusCode":200,"responseTime":109,"contentLength":9},"message":"POST /elasticsearch/logstash-/_field_stats?level=indices 200 109ms - 9.0B"}

When I start kibana, it will be running for sometime. But within seconds status will change to not running. I don't understand what is happening behind. Previously logserver was working perfectly, but once I stop and start the aws instance this issue came.
Please help me to solve.


(Tim Birkett) #2

What are the permissions on the file /opt/kibana/optimize/.babelcache.json?


ls -l /opt/kibana/optimize/.babelcache.json

and paste the output.

Also, why are you running kibana with sudo? Running it with root privileges is probably not ideal.

(Babeesh) #3

-rwxr-xr-x 1 kibana kibana 2 Feb 26 10:21 /opt/kibana/optimize/.babelcache.json

Without sudo I got the message I put under ./bin/kibana
So I tried with sudo. Also with sudo , the server status page will load, and it will show as waiting for plugin elasticsearch, and after few seconds "Request Timout " message will come.

(Babeesh) #4

I am here sharing elasticsearch log also,,,

at org.elasticsearch.index.shard.StoreRecoveryService$
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$
[2016-02-25 11:19:32,106][WARN ][index.engine ] [ontash_prod] [logstash-2016.02.24][2] failed engine [create] No space left on device
at Method)

(Tim Birkett) #5

That probably explains it... No space left on device

Do you have a full disk?

(Tim Birkett) #6

If you can access the Elasticsearch HTTP API you can delete old indexes with a curl command. Something like:

curl -XDELETE http://localhost:9200/logstash-2016.01.*

will remove all logs from January (2016.01).

Take a look at Curator for managing indexes (deleting or moving old data).

(Babeesh) #7

I think space issue is the problem, I chkecked instance system information, it gives
System load: 0.0 Memory usage: 2% Processes: 80
Usage of /: 96.2% of 29.39GB Swap usage: 0% Users logged in: 0
So I will try deleting logs as you said. And will let you know the result.

(Babeesh) #8

I uninstalled elasticsearch. will start from begining. But still system is using 76% of memory.
Is there any hidden folders or something which keep old logs??

(Tim Birkett) #9

DId you manage to get to the bottom of this? I'm not sure of your configuration (volumes / data storage configuration).

(Babeesh) #10


I removed elasticsearch completely and reinstalled it. Configured elasticsearch and kibana correctly. It works .
I am trying to move logs to aws s3 to save disk space in ec2.


(system) #11