ES Request Timeout after 30000ms

Elastic Stack Kibana
1

I've seem to have lost connection from kibana 4.3 to ES 2.1 while idling over xmas, wondering how to resolve this as ES says:

[root@d1r1n1 ~]# curl -XGET "http://`hostname`:9200/_cluster/health?pretty"
{
  "cluster_name" : "mxes_data",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 14,
  "number_of_data_nodes" : 14,
  "active_primary_shards" : 141,
  "active_shards" : 282,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Kibana stdout shows this when restarted:

{"type":"log","@timestamp":"2016-01-01T19:19:59+00:00","tags":["listening","info"],"pid":19650,"message":"Server running at https://10.45.70.62:8080"}
{"type":"log","@timestamp":"2016-01-01T19:20:29+00:00","tags":["status","plugin:elasticsearch","error"],"pid":19650,"name":"plugin:elasticsearch","state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

but I can see that kibana establishes a connection to ES which is renewed at intervals:

[root@d1r1n1 ~]# netstat -atp | grep kibana
tcp 0 25053 d1r1n1.nat.tdcfoo:elastic kibana.mxi.tdcfoo:35442 ESTABLISHED 7608/java
tcp 0 42950 d1r1n1.nat.tdcfoo:elastic kibana.mxi.tdcfoo:35426 LAST_ACK -
[root@d1r1n1 ~]# netstat -atp | grep kibana
tcp 0 42949 d1r1n1.nat.tdcfoo:elastic kibana.mxi.tdcfoo:35446 ESTABLISHED 7608/java
tcp 0 42950 d1r1n1.nat.tdcfoo:elastic kibana.mxi.tdcfoo:35426 LAST_ACK -
[root@d1r1n1 ~]# ping d1r1n1.nat.tdcfoo
PING d1r1n1.nat.tdcfoo (10.45.70.101) 56(84) bytes of data.
64 bytes from d1r1n1.nat.tdcfoo (10.45.70.101): icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from d1r1n1.nat.tdcfoo (10.45.70.101): icmp_seq=2 ttl=64 time=0.010 ms
64 bytes from d1r1n1.nat.tdcfoo (10.45.70.101): icmp_seq=3 ttl=64 time=0.012 ms
64 bytes from d1r1n1.nat.tdcfoo (10.45.70.101): icmp_seq=4 ttl=64 time=0.012 ms
64 bytes from d1r1n1.nat.tdcfoo (10.45.70.101): icmp_seq=5 ttl=64 time=0.010 ms
64 bytes from d1r1n1.nat.tdcfoo (10.45.70.101): icmp_seq=6 ttl=64 time=0.011 ms
64 bytes from d1r1n1.nat.tdcfoo (10.45.70.101): icmp_seq=7 ttl=64 time=0.010 ms
64 bytes from d1r1n1.nat.tdcfoo (10.45.70.101): icmp_seq=8 ttl=64 time=0.009 ms
64 bytes from d1r1n1.nat.tdcfoo (10.45.70.101): icmp_seq=9 ttl=64 time=0.009 ms
^C
--- d1r1n1.nat.tdcfoo ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 8961ms
rtt min/avg/max/mdev = 0.009/0.010/0.012/0.002 ms

2

Is there anything in your ES logs at the same time?

3

No, nothing indicate that ES isn't working, I can see status,state and browse data + indices through ES:plugin:head/kopf and I see that new data/docs are been inserted from beats+logstash instances

4

Finally figured it out. It turned out one of our hypervisor nodes, where my kibana VM was running somehow reverted an openvswitch to default mtu of 1500 instead of 9000, which kibana VM used, hence tcp replies never came back from ES nodes which also are using mtu of 9000 :slight_smile:

5

Im having a very similar issue @stefws . Im getting:
[error][status][plugin:elasticsearch] Status changed from green to red - Request Timeout after 1500ms

How did you solve this?

6

Per his last post, it was an MTU issue.

7

Thanks @warkolm . I found the commit in which this is fixed:

https://github.com/spalger/kibana/commit/dda1217a7fcb5c9bdf05cf7b3f9c29bf95424181

Essentially there is a request_timeout setting in config/kibana.yml

However this still doesn't solve the problem for me :disappointed_relieved:

How do I edit MTU?

This is my log.. it connects and disconnects intermitedly:

15:16:53.266] [error][status][plugin:elasticsearch] Status changed from green to red - Request Timeout after 30000ms
log [15:16:57.278] [error][status][plugin:elasticsearch] Status changed from red to red - Request Timeout after 1500ms
log [15:17:01.652] [info][status][plugin:elasticsearch] Status changed from red to green - Kibana index ready
log [15:17:06.224] [error][status][plugin:elasticsearch] Status changed from green to red - Request Timeout after 1500ms
log [15:17:39.414] [error][status][plugin:elasticsearch] Status changed from red to red - Request Timeout after 30000ms
log [15:17:43.818] [info][status][plugin:elasticsearch] Status changed from red to green - Kibana index ready
log [15:17:48.393] [error][status][plugin:elasticsearch] Status changed from green to red - Request Timeout after 1500ms
log [15:18:21.504] [error][status][plugin:elasticsearch] Status changed from red to red - Request Timeout after 30000ms