Kibana 5.6.6 Crashing After 30s

Elastic Stack Kibana
1

After launching Kibana 5.6.6 it appears to crash unexpectedly. I can perform a couple of searches before it times out. We attempted to increase the timeout rate in the config file but that did not appear to help any suggestions would be greatly appreciated:

  log   [14:51:27.420] [error][status][plugin:elasticsearch@5.6.6] Status changed from green to red - Request Timeout after 3000ms
  log   [14:51:27.423] [error][status][ui settings] Status changed from green to red - Elasticsearch plugin is red
  log   [14:52:17.170] [error][admin][elasticsearch] Request error, retryingGET http://x.x.x.x:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => socket hang up
  log   [14:52:47.170] [error][status][plugin:elasticsearch@5.6.6] Status changed from red to red - Request Timeout after 60000ms
  log   [14:52:49.359] [error][admin][elasticsearch] Request error, retryingPOST http://x.x.x.x:9200/.kibana/_msearch => socket hang up
  log   [14:53:19.363] [warning][admin][elasticsearch] Unable to revive connection: http://x.x.x.x:9200/
  log   [14:53:19.675] [error][admin][elasticsearch] Request error, retryingGET http://x.x.x.x:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => socket hang up
  log   [14:53:49.678] [warning][admin][elasticsearch] Unable to revive connection: http://x.x.x.x:9200/
  log   [14:54:22.178] [warning][admin][elasticsearch] Unable to revive connection: http://x.x.x.x:9200/
  log   [14:54:22.179] [warning][admin][elasticsearch] No living connections  log   [14:54:22.181] [error][status][plugin:elasticsearch@5.6.6] Status changed from red to red - No Living connections
  log   [14:54:54.685] [warning][admin][elasticsearch] Unable to revive connection: http://x.x.x.x:9200/
  log   [14:54:54.686] [warning][admin][elasticsearch] No living connections  log   [14:55:27.190] [warning][admin][elasticsearch] Unable to revive connection: http://x.x.x.x:9200/
  log   [14:55:27.192] [warning][admin][elasticsearch] No living connections  log   [14:55:59.698] [warning][admin][elasticsearch] Unable to revive connection: http://x.x.x.x:9200/
  log   [14:55:59.699] [warning][admin][elasticsearch] No living connections  log   [14:56:32.202] [warning][admin][elasticsearch] Unable to revive connection: http://x.x.x.x:9200/

Also seeing this in Kibana UI:

Error: Request Timeout after 60000ms
    at http://x.x.x.x:5601/bundles/kibana.bundle.js?v=15605:13:4431
    at http://x.x.x.x:5601/bundles/kibana.bundle.js?v=15605:13:4852
2

@tylersmalley or @jbudz - can shed some light here. I tried spinning a cloud 5.6 cluster and it works without a crash as mentioned here.

Thanks
Rashmi

3

It's not Kibana that is crashing, it's that Kibana is unable to reach Elasticsearch.

Can you check the Cluster Health Elasticsearch API from the server Kibana is on?

(replacing x.x.x.x with your IP)

curl -X GET "http://x.x.x.x:9200/_cluster/health&pretty"

4

Tyler,

The cluster health is green but as soon as I launch Kibana it wont stay up long once we start running searches. I have another UI to search elastic that is SQL based and I never have any issues but as soon as I start using Kibana and begin searching is when the issues begin.

{
"cluster_name": "MY-CLUSTER01",
"status": "green",
"timed_out": false,
"number_of_nodes": 9,
"number_of_data_nodes": 9,
"active_primary_shards": 300,
"active_shards": 600,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100.0
}

5

What plugins do you have installed?

6

None that I'm aware of.

7

Is Kibana hitting Elasticsearch directly, or do you have a load balancer or anything between? I would check that and the Elasticsearch logs. Kibana is getting a socket timeout on that connection so no much I can think to do on the Kibana side.

8

Kibana is hitting Elastic directly both are on separate boxes but they are on the same subnet. No load balancer between them.

9

Is this taking down the Elasticsearch cluster? Once this happens, what does the _cluster/health look like then?

10

When we start running intensive searches I am unable to use postman or curl on the server to get cluster health its almost like the indexer is so busy it cant even run the cluster health command: localhost:9200/_cluster/health. I was alerted through a third party app that the cluster state is critical so I would assume it is in a Red state right now. Any suggestions on what could be causing this?

11

Update: I did confirm that the cluster did go red we are still in a yellow state but expect to go back to green soon.

12

Thanks for the update . Keep us posted.

13

The problem is Kibana wont stay up without the cluster going red. However we would like to use Kibana without it crashing. What could we do to make sure it remains stable when performing searches on it? What I'm think is happening is that when we run a large search its opening closed indecies and causes the JVM to crash...

14

@rashmi or @tylersmalley any suggestions?

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.