Loss of connection, Timeout

Elastic Stack Kibana
1

After a while (couldn't tell how long), Kibana loses connection to the ES cluster (tested with two installations, one locally on the ELK stack, one from a remote machine. I am getting these error messages:

2017/02/28 12:51:25 warn 19741#0: *335 upstream server temporarily disabled while connecting to upstream, client: 192.168.1.120, server: n2, request: "GET /bundles/kibana.style.css?v=9910 HTTP/1.1", upstream: "http://[::1]:5601/bundles/kibana.style.css?v=9910", host: "XXXXX:8888", referrer: "http://XXXXX:8888/app/kibana"

2017/02/28 12:51:57 [info] 19741#0: 332 kevent() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: 192.168.1.120, server: n2, request: "POST /elasticsearch/filebeat-/_field_stats?level=indices HTTP/1.1", upstream: "http://127.0.0.1:5601/elasticsearch/filebeat-*/_field_stats?level=indices", host: "XXXXX:8888", referrer: "http://XXXXX:8888/app/kibana"

where 192.168.1.120 is the PC I'm on, and XXXXX is an OpenBSD6.0 machine that's forwarding Port 8888 ro localhost:5601.

Restarting Kibana is giving me access again.

Where should I begin to dig?

TIA
Matthias

2

@mkoch which version of Kibana/ES are you running?

3

It's Elasticsearch 2.4.1 and Kibana 4.4.2 (on a CentOS7 ELK stack) and Kibana 4.5.3 on the OpenBSD machine. Kibana behaves the same way on both machines.

4

@mkoch when you get this error, are you able to access Kibana directly without going through the proxy?

5

I'll be watching it. Recently I've had the impression that the problem is somehow related to idle times. I don't think it ever occurred during sessions with short autorefresh periods. Reconfigured it now to make it directly accessible through port 5601 and bypass nginx.

6

It has just happened again - after having been idle for 18 hours, both Kibana instances failed. For clarification: I have two Kibana instances, one is sitting on top of the ELK stack (running on the same machine as ES and logstash); the other is on a dedicated machine; both are accessible through nginx and directly (port 5601).

Both machines threw the "timeout after 30 seconds" error, both through nginx and directly. But this time they reappeared without me having to restart Kibana. This probably means that not Kibana, but the ES cluster is the culprit, as two independent Kibana machines with differens OSs timed out.

7

But then, I never restart ES, just Kibana. ES seems to be doing fine.

[xxxxx@xxxxx ~]# curl -XGET http://localhost:9200/_cluster/health?pretty {
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 861,
"active_shards" : 1391,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.