# Elasticsearch causing Kibana flapping -- frequent time outs **URL:** \ **Category:** Elasticsearch\ **Created:** [November 30, 2016, 6:24pm UTC](https://discuss.elastic.co/t/elasticsearch-causing-kibana-flapping-frequent-time-outs/67661 "2016-11-30T18:24:01Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![john\_czerwiec](https://avatars.discourse-cdn.com/v4/letter/j/96bed5/32.png) [@john\_czerwiec](https://discuss.elastic.co/u/john_czerwiec)\ **Post date:** [November 30, 2016, 6:24pm UTC](https://discuss.elastic.co/t/elasticsearch-causing-kibana-flapping-frequent-time-outs/67661/1 "2016-11-30T18:24:01Z")
![](https://us1.discourse-cdn.com/elastic/original/2X/6/67e3a46d39e168753b94e153a699c7549fb2a559.png) Cluster includes 4 nodes that are on rather robust AWS instance (8 cores, 64G ram). Kibana is on same host as es master, three other nodes are separate. (snipets:) * * * ``` kibana:~$ curl -XGET 'http://localhost:9200/_cat/nodes?h=host,role,v,rc,hc,hp,heapMax,fielddataMemory,idti,iiti,mtt.rtisqti' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 228 0 228 0 0 0 0 --:--:-- 0:04:42 --:--:-- 66 ``` Why would this take so long??? ``` 10.60.20.31 d 2.3.5 31.2gb 12.4gb 83 14.9gb 0b 0s 6.4h 10.60.20.76 - 2.3.5 24.5gb 6.7gb 44 14.9gb 0b 0s 0s 10.60.20.25 d 2.3.5 31.1gb 14.6gb 98 14.9gb 0b 0s 5.3h 10.60.20.180 d 2.3.5 27.5gb 14.2gb 95 14.9gb 0b 0s 2h ``` * * * ``` **kibana:~$ docker logs --tail=5000 1ac66b2c5da0 |grep timed** ReceiveTimeoutTransportException[[][10.60.20.31:9300][internal:discovery/zen/unicast] request_id [841513] timed out after [3750ms]] ReceiveTimeoutTransportException[[][10.60.20.31:9300][internal:discovery/zen/unicast] request_id [841516] timed out after [3750ms]] ``` ... ReceiveTimeoutTransportException[[10.60.20.31:9300][inter[2016-11-17 16:03:45,635][WARN][transport] [kibana] Received response for a request that has timed out, sent [37828ms] ago, timed out [7828ms] ago, action [internal:discovery/zen/fd/master\_ping], node [{es100}{TOxTFYZmRDGIj8IUMg2lfg}{10.60.20.31}{10.60.20.31:9300}], id [842740] ``` [2016-11-22 14:50:01,936][WARN][transport] [kibana] Received response for a request that has timed out, sent [33223ms] ago, timed out [3222ms] ago, action [internal:discovery/zen/fd/master_ping], node [{es100}{ZaUOdCn3SQ2nxl2LEgHjMQ}{10.60.20.31}{10.60.20.31:9300}], id [2250455] ... ---------- **~es100:~$ docker logs --tail=5000 1f1a3fa61a24 |grep timed** ReceiveTimeoutTransportException[[es101][10.60.20.25:9300][cluster:monitor/nodes/stats[n]] request_id [3021166] timed out after [15000ms]] [2016-11-30 15:10:22,508][WARN][discovery.zen.publish] [es100] timed out waiting for all nodes to process published state [21309] (timeout [30s], pending nodes: [{es101}{QTS9MboWT8qWuqAa-e8Slw}{10.60.20.25}{10.60.20.25:9300}]) ``` ... `[2016-11-30 15:23:41,705][WARN][discovery.zen.publish] [es100] timed out waiting for all nodes to process published state [21358] (timeout [30s], pending nodes: [{es102}{ZWdHUPOlRIOSzppyKCYjMA}{10.60.20.180}{10.60.20.180:9300}, {es101}{QTS9MboWT8qWuqAa-e8Slw}{10.60.20.25}{10.60.20.25:9300}])` ``` ---------- **~es101:~$ docker logs --tail=2000 0adf150c763c |grep timed** [2016-11-30 16:01:17,143][WARN][transport] [es101] Received response for a request that has timed out, sent [212728ms] ago, timed out [163887ms] ago, action [internal:discovery/zen/fd/master_ping], node [{es100}{xDmQcVNFSgaUo2yicoXztA}{10.60.20.31}{10.60.20.31:9300}], id [1471663] [2016-11-30 16:01:17,143][WARN][transport] [es101] Received response for a request that has timed out, sent [163887ms] ago, timed out [126726ms] ago, action [internal:discovery/zen/fd/master_ping], node [{es100}{xDmQcVNFSgaUo2yicoXztA}{10.60.20.31}{10.60.20.31:9300}], id [1471961] [2016-11-30 16:38:39,771][WARN][transport] [es101] Received response for a request that has timed out, sent [50867ms] ago, timed out [8ms] ago, action [internal:discovery/zen/unicast], node [{#zen_unicast_1#}{10.60.20.31}{10.60.20.31:9300}], id [1473141] ---------- **es102:~$ docker logs --tail=5000 b46967bafaa9 |grep timed** [2016-11-30 16:51:19,567][WARN][discovery.zen.publish] [es102] timed out waiting for all nodes to process published state [21609] (timeout [30s], pending nodes: [{es101}{QTS9MboWT8qWuqAa-e8Slw}{10.60.20.25}{10.60.20.25:9300}]) ReceiveTimeoutTransportException[[es101][10.60.20.25:9300][cluster:monitor/nodes/stats[n]] request_id [6330844] timed out after [15000ms]] [2016-11-30 16:55:26,395][WARN][discovery.zen.publish] [es102] timed out waiting for all nodes to process published state [21625] (timeout [30s], pending nodes: [{es101}{QTS9MboWT8qWuqAa-e8Slw}{10.60.20.25}{10.60.20.25:9300}]) ``` ... `[2016-11-30 18:03:44,950][WARN][discovery.zen.publish] [es102] timed out waiting for all nodes to process published state [21782] (timeout [30s], pending nodes: [{es101}{QTS9MboWT8qWuqAa-e8Slw}{10.60.20.25}{10.60.20.25:9300}])` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 28, 2016, 6:24pm UTC](https://discuss.elastic.co/t/elasticsearch-causing-kibana-flapping-frequent-time-outs/67661/2 "2016-12-28T18:24:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.