Elasticsearch causing Kibana flapping -- frequent time outs


(John Czerwiec) #1

Cluster includes 4 nodes that are on rather robust AWS instance (8 cores, 64G ram). Kibana is on same host as es master, three other nodes are separate.

(snipets:)


kibana:~$ curl -XGET 'http://localhost:9200/_cat/nodes?h=host,role,v,rc,hc,hp,heapMax,fielddataMemory,idti,iiti,mtt.rtisqti'
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   228    0   228    0     0      0      0 --:--:--  0:04:42 --:--:--    66

Why would this take so long???

10.60.20.31  d 2.3.5 31.2gb 12.4gb 83 14.9gb 0b 0s 6.4h
10.60.20.76  - 2.3.5 24.5gb  6.7gb 44 14.9gb 0b 0s   0s
10.60.20.25  d 2.3.5 31.1gb 14.6gb 98 14.9gb 0b 0s 5.3h
10.60.20.180 d 2.3.5 27.5gb 14.2gb 95 14.9gb 0b 0s   2h

**kibana:~$ docker logs --tail=5000 1ac66b2c5da0   |grep timed**
ReceiveTimeoutTransportException[[][10.60.20.31:9300][internal:discovery/zen/unicast] request_id [841513] timed out after [3750ms]]
ReceiveTimeoutTransportException[[][10.60.20.31:9300][internal:discovery/zen/unicast] request_id [841516] timed out after [3750ms]]

...
ReceiveTimeoutTransportException[[][10.60.20.31:9300][inter[2016-11-17 16:03:45,635][WARN ][transport ] [kibana] Received response for a request that has timed out, sent [37828ms] ago, timed out [7828ms] ago, action [internal:discovery/zen/fd/master_ping], node [{es100}{TOxTFYZmRDGIj8IUMg2lfg}{10.60.20.31}{10.60.20.31:9300}], id [842740]

        [2016-11-22 14:50:01,936][WARN ][transport                ] [kibana] Received response for a request that has timed out, sent [33223ms] ago, timed out [3222ms] ago, action [internal:discovery/zen/fd/master_ping], node [{es100}{ZaUOdCn3SQ2nxl2LEgHjMQ}{10.60.20.31}{10.60.20.31:9300}], id [2250455]
...

----------
**~es100:~$  docker logs --tail=5000 1f1a3fa61a24 |grep timed**
ReceiveTimeoutTransportException[[es101][10.60.20.25:9300][cluster:monitor/nodes/stats[n]] request_id [3021166] timed out after [15000ms]]
[2016-11-30 15:10:22,508][WARN ][discovery.zen.publish    ] [es100] timed out waiting for all nodes to process published state [21309] (timeout [30s], pending nodes: [{es101}{QTS9MboWT8qWuqAa-e8Slw}{10.60.20.25}{10.60.20.25:9300}])

...
[2016-11-30 15:23:41,705][WARN ][discovery.zen.publish ] [es100] timed out waiting for all nodes to process published state [21358] (timeout [30s], pending nodes: [{es102}{ZWdHUPOlRIOSzppyKCYjMA}{10.60.20.180}{10.60.20.180:9300}, {es101}{QTS9MboWT8qWuqAa-e8Slw}{10.60.20.25}{10.60.20.25:9300}])

----------  
**~es101:~$  docker logs --tail=2000 0adf150c763c  |grep timed** 
    [2016-11-30 16:01:17,143][WARN ][transport                ] [es101] Received response for a request that has timed out, sent [212728ms] ago, timed out [163887ms] ago, action [internal:discovery/zen/fd/master_ping], node [{es100}{xDmQcVNFSgaUo2yicoXztA}{10.60.20.31}{10.60.20.31:9300}], id [1471663]
    [2016-11-30 16:01:17,143][WARN ][transport                ] [es101] Received response for a request that has timed out, sent [163887ms] ago, timed out [126726ms] ago, action [internal:discovery/zen/fd/master_ping], node [{es100}{xDmQcVNFSgaUo2yicoXztA}{10.60.20.31}{10.60.20.31:9300}], id [1471961]
 
    [2016-11-30 16:38:39,771][WARN ][transport                ] [es101] Received response for a request that has timed out, sent [50867ms] ago, timed out [8ms] ago, action [internal:discovery/zen/unicast], node [{#zen_unicast_1#}{10.60.20.31}{10.60.20.31:9300}], id [1473141]


----------

**es102:~$ docker logs --tail=5000 b46967bafaa9  |grep timed**
[2016-11-30 16:51:19,567][WARN ][discovery.zen.publish    ] [es102] timed out waiting for all nodes to process published state [21609] (timeout [30s], pending nodes: [{es101}{QTS9MboWT8qWuqAa-e8Slw}{10.60.20.25}{10.60.20.25:9300}])

ReceiveTimeoutTransportException[[es101][10.60.20.25:9300][cluster:monitor/nodes/stats[n]] request_id [6330844] timed out after [15000ms]]
[2016-11-30 16:55:26,395][WARN ][discovery.zen.publish    ] [es102] timed out waiting for all nodes to process published state [21625] (timeout [30s], pending nodes: [{es101}{QTS9MboWT8qWuqAa-e8Slw}{10.60.20.25}{10.60.20.25:9300}])

...
[2016-11-30 18:03:44,950][WARN ][discovery.zen.publish ] [es102] timed out waiting for all nodes to process published state [21782] (timeout [30s], pending nodes: [{es101}{QTS9MboWT8qWuqAa-e8Slw}{10.60.20.25}{10.60.20.25:9300}])


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.