# Kibana showing "Red" status, need help diagnosing/fixing

**URL:** <https://discuss.elastic.co/t/kibana-showing-red-status-need-help-diagnosing-fixing/93432>\
**Category:** Kibana\
**Created:** [July 17, 2017, 5:09pm UTC](https://discuss.elastic.co/t/kibana-showing-red-status-need-help-diagnosing-fixing/93432 "2017-07-17T17:09:19Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![wdennis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wdennis/32/20119_2.png) [@wdennis](https://discuss.elastic.co/u/wdennis)\
**Post date:** [July 17, 2017, 5:09pm UTC](https://discuss.elastic.co/t/kibana-showing-red-status-need-help-diagnosing-fixing/93432/1 "2017-07-17T17:09:19Z")

</div>

Hi all, ELK n00b here... I had a previously-working Kibana, but now it is in "Red" status, and I don't know how to go about diagnosing/fixing this... I will provide some details, and could someone kindly point me in the right direction 🙂

First off, running ELK stack version 5.4.1 on Ubuntu 16.04.2, all on a single server --

```
root@logstash01:/var/log# dpkg -l | grep -e elastic -e logstash -e kibana
    ii elasticsearch 5.4.1 all Elasticsearch is a distributed RESTful ...
    ii kibana 5.4.1 amd64 Explore and visualize your Elasticsearch data
    ii logstash 1:5.4.1-1 all An extensible logging pipeline

```

I'm seeing this on the Kibana console:

 ![](https://us1.discourse-cdn.com/elastic/original/3X/2/b/2baf748aad97ebce8ae737820a42cd8a9c8b9310.png)

And when I do a GET on status:

```
root@logstash01:/var/log# curl -XGET 'http://localhost:9200/_cluster/health?pretty'
{
  "cluster_name" : "logstash",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 235,
  "active_shards" : 235,
  "relocating_shards" : 0,
  "initializing_shards" : 4,
  "unassigned_shards" : 2683,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 6,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 178,
  "active_shards_percent_as_number" : 8.042436687200547
}

```

Where do I go from here?

---

<div class="post-metadata">

**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\
**Post date:** [July 17, 2017, 6:13pm UTC](https://discuss.elastic.co/t/kibana-showing-red-status-need-help-diagnosing-fixing/93432/2 "2017-07-17T18:13:17Z")

</div>

Hi Willard

That health query you did shows Elasticsearch status is red. I also see you have initializing\_shards. That should be a short temporary condition which should clear up. If it doesn't, you should probably check the Elasticsearch log. It's probably at  
`/var/log/elasticsearch/elasticsearch.log`

If the problem isn't obvious, you should probably post a question on the Elasticsearch forum.

Thanks,  
Lee

---

<div class="post-metadata">

**Author:** ![wdennis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wdennis/32/20119_2.png) [@wdennis](https://discuss.elastic.co/u/wdennis)\
**Post date:** [July 17, 2017, 6:15pm UTC](https://discuss.elastic.co/t/kibana-showing-red-status-need-help-diagnosing-fixing/93432/3 "2017-07-17T18:15:39Z")

</div>

Thanks @LeeDr - I do have some sort of problem going on w/ Elasticsearch...

```
root@logstash01:/var/log# systemctl status elasticsearch
    * elasticsearch.service - Elasticsearch
       Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Mon 2017-07-17 13:18:12 EDT; 54min ago
         Docs: http://www.elastic.co
      Process: 5308 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -E
      Process: 5304 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0 Main PID: 5308 (code=exited, status=127)

Jul 17 13:07:13 logstash01 systemd[1]: Starting Elasticsearch...
Jul 17 13:07:13 logstash01 systemd[1]: Started Elasticsearch.
Jul 17 13:18:12 logstash01 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=127/n/a Jul 17 13:18:12 logstash01 systemd[1]: elasticsearch.service: Unit entered failed state.
Jul 17 13:18:12 logstash01 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

```

Will check the logs, and proceed from there...

---

<div class="post-metadata">

**Author:** ![niteshkumar](https://avatars.discourse-cdn.com/v4/letter/n/f05b48/32.png) [@niteshkumar](https://discuss.elastic.co/u/niteshkumar)\
**Post date:** [July 29, 2017, 12:32pm UTC](https://discuss.elastic.co/t/kibana-showing-red-status-need-help-diagnosing-fixing/93432/4 "2017-07-29T12:32:05Z")

</div>

Hi There. I am also facing the same issue. My Kibana was working earlier but not now.

 ![01 PM](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bf7003a8f509c10040e78aeff224546f74e334b8.png)

[root@ip-10-254-10-58 ~]# curl -XGET '[http://localhost:9200/\_cluster/health?pretty](http://localhost:9200/_cluster/health?pretty)'  
{  
"cluster\_name" : "elasticsearch",  
"status" : "red",  
"timed\_out" : false,  
"number\_of\_nodes" : 1,  
"number\_of\_data\_nodes" : 1,  
"active\_primary\_shards" : 0,  
"active\_shards" : 0,  
"relocating\_shards" : 0,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 0,  
"delayed\_unassigned\_shards" : 0,  
"number\_of\_pending\_tasks" : 0,  
"number\_of\_in\_flight\_fetch" : 0,  
"task\_max\_waiting\_in\_queue\_millis" : 0,  
"active\_shards\_percent\_as\_number" : "NaN"  
}

Kibana log:  
{"type":"response","@timestamp":"2017-07-29T12:21:20Z","tags":,"pid":3265,"method":"get","statusCode":304,"req":{"url":"/ui/favicons/favicon-16x16.png","method":"get","headers":{"host":"[prodsharedelk01.cloud.operative.com:5601](http://prodsharedelk01.cloud.operative.com:5601)","connection":"keep-alive","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_12\_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36","accept":"image/webp,image/apng,image/_,_/\*;q=0.8","referer":"[http://prodsharedelk01.cloud.operative.com:5601/app/kibana","accept-encoding":"gzip](http://prodsharedelk01.cloud.operative.com:5601/app/kibana%22,%22accept-encoding%22:%22gzip), deflate","accept-language":"en-US,en;q=0.8","if-none-match":""f33f077bfe13045136046c93b6180be0379386ff"","if-modified-since":"Tue, 06 Dec 2016 13:06:55 GMT"},"remoteAddress":"10.111.7.223","userAgent":"10.111.7.223","referer":"[http://prodsharedelk01.cloud.operative.com:5601/app/kibana"},"res":{"statusCode":304,"responseTime":0,"contentLength":9},"message":"GET](http://prodsharedelk01.cloud.operative.com:5601/app/kibana%22%7D,%22res%22:%7B%22statusCode%22:304,%22responseTime%22:0,%22contentLength%22:9%7D,%22message%22:%22GET) /ui/favicons/favicon-16x16.png 304 0ms - 9.0B"}  
{"type":"log","@timestamp":"2017-07-29T12:23:47Z","tags":["error","elasticsearch"],"pid":3265,"message":"Request error, retrying\nHEAD [http://localhost:9200/](http://localhost:9200/) =\> read ECONNRESET"}  
{"type":"log","@timestamp":"2017-07-29T12:23:47Z","tags":["warning","elasticsearch"],"pid":3265,"message":"Unable to revive connection: [http://localhost:9200/](http://localhost:9200/)"}  
{"type":"log","@timestamp":"2017-07-29T12:23:47Z","tags":["warning","elasticsearch"],"pid":3265,"message":"No living connections"}  
{"type":"log","@timestamp":"2017-07-29T12:23:47Z","tags":["status","plugin:elasticsearch@5.1.1","error"],"pid":3265,"state":"red","message":"Status changed from red to red - Unable to connect to Elasticsearch at [http://localhost:9200](http://localhost:9200).","prevState":"red","prevMsg":"Request Timeout after 3000ms"}

Elasticsearch logs  
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s  
at org.elasticsearch.cluster.service.ClusterService.lambda$null$4(ClusterService.java:449) ~[elasticsearch-5.1.1.jar:5.1.1]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) ~[elasticsearch-5.1.1.jar:5.1.1]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0\_121]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0\_121]  
at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_121]  
[2017-07-29T08:29:04,686][DEBUG][o.e.a.b.TransportShardBulkAction] [Na2ijff] [filebeat-2017.06.29][1] failed to execute bulk item (index) index {[filebeat-2017.06.29][springboot][AV2OUdXlwentso5\_JxOS], source[{"offset":47884739,"level":"INFO","input\_type":"log","logmessage":"Response received from Connect-Platform.","pid":"25874","source":"/opt/operative/connect-web/log/conw.log","thread":"tp1126112943-41","message":"2017-06-29 11:38:56.053 INFO 25874 --- [tp1126112943-41] c.o.gateway.handler.ProposalHandler : Response received from Connect-Platform.","type":"springboot","tags":["stg","name:stgconnectweb02","id:connect","role:webserver","id:stg","type:","beats\_input\_codec\_multiline\_applied"],"@timestamp":"2017-06-29T15:38:56.053Z","@version":"1","beat":{"hostname":"stgconnectweb02","name":"stgconnectweb02","version":"5.1.1"},"host":"stgconnectweb02","class":"ProposalHandler","timestamp":"2017-06-29 11:38:56.053"}]}  
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s  
at org.elasticsearch.cluster.service.ClusterService.lambda$null$4(ClusterService.java:449) ~[elasticsearch-5.1.1.jar:5.1.1]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) ~[elasticsearch-5.1.1.jar:5.1.1]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0\_121]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0\_121]  
at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_121]  
[2017-07-29T08:29:18,413][INFO][o.e.c.m.MetaDataMappingService] [Na2ijff] [filebeat-2017.06.29/TDDk5G65S560i4Qfm0Ln3w] create\_mapping [springboot]

---

<div class="post-metadata">

**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\
**Post date:** [August 1, 2017, 1:06am UTC](https://discuss.elastic.co/t/kibana-showing-red-status-need-help-diagnosing-fixing/93432/5 "2017-08-01T01:06:52Z")

</div>

Hi niteshkumar,  
That looks like a good question for the Elasticsearch forum. Please include the Elasticsearch version, and Operating System. Maybe also the Java info.

You might also check this post;

> [@\[ElasticSearch 2.2.0\] I am occasionally getting Process Cluster Event Timeout Exception\[failed to process cluster event (put-mapping \[as\]) within 30s\] while bulk indexing documents](https://discuss.elastic.co/t/elasticsearch-2-2-0-i-am-occasionally-getting-process-cluster-event-timeout-exception-failed-to-process-cluster-event-put-mapping-as-within-30s-while-bulk-indexing-documents/42305/6):
>
> Would this be the same reason why the cluster throws ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [as]) within 30s]?

Which then leads to this;

> [@What happends if I change master: true, data: true node to master-only node?](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/16):
>
> Force-deleting half of index directories didn't help making the cluster back alive. I tried doubling the master node's memory, but it still dies. My initial plan was to make the cluster back alive and re-indexing everything. (with my-index-YYYYMM pattern) Since the cluster doesn't easily get back on, I plan to create new cluster and find a way to restore data from the dead cluster. (Not sure this is possible at the moment.)

Regards,  
Lee

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 29, 2017, 1:07am UTC](https://discuss.elastic.co/t/kibana-showing-red-status-need-help-diagnosing-fixing/93432/6 "2017-08-29T01:07:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.