Kibana showing "Red" status, need help diagnosing/fixing

[wdennis (Willard Dennis)]

Hi all, ELK n00b here... I had a previously-working Kibana, but now it is in "Red" status, and I don't know how to go about diagnosing/fixing this... I will provide some details, and could someone kindly point me in the right direction

First off, running ELK stack version 5.4.1 on Ubuntu 16.04.2, all on a single server --

root@logstash01:/var/log# dpkg -l | grep -e elastic -e logstash -e kibana
    ii  elasticsearch                      5.4.1                                      all          Elasticsearch is a distributed RESTful ...
    ii  kibana                             5.4.1                                      amd64        Explore and visualize your Elasticsearch data
    ii  logstash                           1:5.4.1-1                                  all          An extensible logging pipeline

I'm seeing this on the Kibana console:

And when I do a GET on status:

root@logstash01:/var/log# curl -XGET 'http://localhost:9200/_cluster/health?pretty'
{
  "cluster_name" : "logstash",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 235,
  "active_shards" : 235,
  "relocating_shards" : 0,
  "initializing_shards" : 4,
  "unassigned_shards" : 2683,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 6,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 178,
  "active_shards_percent_as_number" : 8.042436687200547
}

Where do I go from here?

[LeeDr (Lee Drengenberg)]

Hi Willard

That health query you did shows Elasticsearch status is red. I also see you have initializing_shards. That should be a short temporary condition which should clear up. If it doesn't, you should probably check the Elasticsearch log. It's probably at
/var/log/elasticsearch/elasticsearch.log

If the problem isn't obvious, you should probably post a question on the Elasticsearch forum.

Thanks,
Lee

[wdennis (Willard Dennis)]

Thanks @LeeDr - I do have some sort of problem going on w/ Elasticsearch...

root@logstash01:/var/log# systemctl status elasticsearch
    * elasticsearch.service - Elasticsearch
       Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Mon 2017-07-17 13:18:12 EDT; 54min ago
         Docs: http://www.elastic.co
      Process: 5308 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -E
      Process: 5304 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0 Main PID: 5308 (code=exited, status=127)

Jul 17 13:07:13 logstash01 systemd[1]: Starting Elasticsearch...
Jul 17 13:07:13 logstash01 systemd[1]: Started Elasticsearch.
Jul 17 13:18:12 logstash01 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=127/n/a Jul 17 13:18:12 logstash01 systemd[1]: elasticsearch.service: Unit entered failed state.
Jul 17 13:18:12 logstash01 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Will check the logs, and proceed from there...

[niteshkumar]

Hi There. I am also facing the same issue. My Kibana was working earlier but not now.

[root@ip-10-254-10-58 ~]# curl -XGET 'http://localhost:9200/_cluster/health?pretty'
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : "NaN"
}

Kibana log:
{"type":"response","@timestamp":"2017-07-29T12:21:20Z","tags":,"pid":3265,"method":"get","statusCode":304,"req":{"url":"/ui/favicons/favicon-16x16.png","method":"get","headers":{"host":"prodsharedelk01.cloud.operative.com:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36","accept":"image/webp,image/apng,image/,/*;q=0.8","referer":"http://prodsharedelk01.cloud.operative.com:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8","if-none-match":""f33f077bfe13045136046c93b6180be0379386ff"","if-modified-since":"Tue, 06 Dec 2016 13:06:55 GMT"},"remoteAddress":"10.111.7.223","userAgent":"10.111.7.223","referer":"http://prodsharedelk01.cloud.operative.com:5601/app/kibana"},"res":{"statusCode":304,"responseTime":0,"contentLength":9},"message":"GET /ui/favicons/favicon-16x16.png 304 0ms - 9.0B"}
{"type":"log","@timestamp":"2017-07-29T12:23:47Z","tags":["error","elasticsearch"],"pid":3265,"message":"Request error, retrying\nHEAD http://localhost:9200/ => read ECONNRESET"}
{"type":"log","@timestamp":"2017-07-29T12:23:47Z","tags":["warning","elasticsearch"],"pid":3265,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2017-07-29T12:23:47Z","tags":["warning","elasticsearch"],"pid":3265,"message":"No living connections"}
{"type":"log","@timestamp":"2017-07-29T12:23:47Z","tags":["status","plugin:elasticsearch@5.1.1","error"],"pid":3265,"state":"red","message":"Status changed from red to red - Unable to connect to Elasticsearch at http://localhost:9200.","prevState":"red","prevMsg":"Request Timeout after 3000ms"}

Elasticsearch logs
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
at org.elasticsearch.cluster.service.ClusterService.lambda$null$4(ClusterService.java:449) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2017-07-29T08:29:04,686][DEBUG][o.e.a.b.TransportShardBulkAction] [Na2ijff] [filebeat-2017.06.29][1] failed to execute bulk item (index) index {[filebeat-2017.06.29][springboot][AV2OUdXlwentso5_JxOS], source[{"offset":47884739,"level":"INFO","input_type":"log","logmessage":"Response received from Connect-Platform.","pid":"25874","source":"/opt/operative/connect-web/log/conw.log","thread":"tp1126112943-41","message":"2017-06-29 11:38:56.053 INFO 25874 --- [tp1126112943-41] c.o.gateway.handler.ProposalHandler : Response received from Connect-Platform.","type":"springboot","tags":["stg","name:stgconnectweb02","id:connect","role:webserver","id:stg","type:","beats_input_codec_multiline_applied"],"@timestamp":"2017-06-29T15:38:56.053Z","@version":"1","beat":{"hostname":"stgconnectweb02","name":"stgconnectweb02","version":"5.1.1"},"host":"stgconnectweb02","class":"ProposalHandler","timestamp":"2017-06-29 11:38:56.053"}]}
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
at org.elasticsearch.cluster.service.ClusterService.lambda$null$4(ClusterService.java:449) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2017-07-29T08:29:18,413][INFO ][o.e.c.m.MetaDataMappingService] [Na2ijff] [filebeat-2017.06.29/TDDk5G65S560i4Qfm0Ln3w] create_mapping [springboot]

[LeeDr (Lee Drengenberg)]

Hi niteshkumar,
That looks like a good question for the Elasticsearch forum. Please include the Elasticsearch version, and Operating System. Maybe also the Java info.

You might also check this post;

[ElasticSearch 2.2.0] I am occasionally getting Process Cluster Event Timeout Exception[failed to process cluster event (put-mapping [as]) within 30s] while bulk indexing documents

Would this be the same reason why the cluster throws ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [as]) within 30s]?

Which then leads to this;

What happends if I change master: true, data: true node to master-only node?

Force-deleting half of index directories didn't help making the cluster back alive. I tried doubling the master node's memory, but it still dies. My initial plan was to make the cluster back alive and re-indexing everything. (with my-index-YYYYMM pattern) Since the cluster doesn't easily get back on, I plan to create new cluster and find a way to restore data from the dead cluster. (Not sure this is possible at the moment.)

Regards,
Lee