# 5.5.3 Node stalls for no apparent reason **URL:** \ **Category:** Elasticsearch\ **Created:** [February 14, 2018, 1:25pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801 "2018-02-14T13:25:04Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 14, 2018, 1:25pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/1 "2018-02-14T13:25:04Z")
We're experiencing an intermittent issue for weeks/months since upgrading to elasticsearch 5.5.3 (from 5.2.x) In short, one or both elasticsearch nodes will suddenly and without no apparent warning stop accepting search requests. In fact it appears as if it literally refuses to do any work at all. The world stops. Thats more than just poetic. Maybe GC is stopping the world?!?!? We run a 2-node cluster (with minimum master nodes set to 1) on Ubuntu 16.04 LTS. Each node running on its own server. Each node has a 16G heap Servers are very capable bare metal Dell blade servers, with SSDs, 180+ GB memory, 80 cpu cores etc. #justsayin We run a few other things on that server and if its relevant I'll disclose but basically the server is way under-utilized all day The outage happens any time of day. Even in the wee hours of the morning when website traffic is at an all-time low. The outage will last 5-20mins. Sometimes the stalled node mysteriously self-recovers and starts accepting requests once more. But more often than not we manually restart the stalled elasticsearch node (the node, not the server). Noteworthy: The Kibana/X-Pack monitoring graph for the stalled node mysteriously goes blank for the duration of the outage. No data. Nothing Here is excerpts of the elasticsearch log file(s). Too large to gist! > **[elastic](https://www.dropbox.com/sh/7qpedo370dt5hpv/AAAx4Sm06owuN5fv1yrOSwEPa?dl=0)** > > Shared with Dropbox app1 is the elastic node that stalls. so its log file is incredibly noisy and verbose app2 is the other node which of course observes the issue from a different but useful viewpoint Some commentary on app1 log: - The outage appears to start around 2018-01-03T18:50 - Some ParsingException's are thrown before that but i think/feel its unrelated - A build up of "monitoring execution is skipped until previous execution terminated" log lines around 2018-01-03T18:51. Seems to suggest something awry? - By 2018-01-03T18:52:36, the outage is in full swing. Numerous EsRejectedExecutionException exceptions thrown Commentary on app2 log: - at 2018-01-03T18:55:08, app2 learns that app1 has left the cluster and it all goes downhill from there We've been running elasticsearch for 3/4 years. Starting with 1.7, then 2.0, then 5.2, and now 5.5.x. Its been stable as a rock until we upgraded to 5.5 We're out of ideas on what the problem may be. Early on we thought the problem might be GC related and so we progressively increased heap size from 8G to 16G but the problem persists We tried googling over and over again to find someone else on the intertubes with the same problem but came up empty Please help!?!?! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 14, 2018, 1:53pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/2 "2018-02-14T13:53:58Z")
If you have 2 nodes that are both master eligible, you should have `minimum_master_nodes` set to `2` [in order to avoid split-brain scenarios](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/modules-node.html#split-brain). Can you change this to see if it makes a difference? ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 14, 2018, 1:59pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/3 "2018-02-14T13:59:02Z")
Thanks for quick reply. Worth trying out ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 14, 2018, 2:35pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/4 "2018-02-14T14:35:38Z")
If you are looking to be able to handle one node going down, you can install one dedicated master node (which can be a lot smaller than the data nodes) on one of the hosts. This will allow a master too be elected even if one of the nodes is not available, which will improve resiliency. If you lose the host with 2 nodes, your cluster will however be in a red state just as with the current configuration. This is why we always recommend 3 master eligible nodes on separate hosts for production. ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 15, 2018, 7:06am UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/5 "2018-02-15T07:06:25Z")
thanks again is adding a 3rd data-only node as simple as setting node.data to false for that particular node? [https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-node.html#data-node](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-node.html#data-node) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 15, 2018, 7:08am UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/6 "2018-02-15T07:08:22Z")
Yes. How to set up a dedicated master node is described [a bit higher up on that page](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-node.html#master-node). ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 16, 2018, 11:09am UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/7 "2018-02-16T11:09:39Z")
Me again We've changed our minimum\_master\_nodes setting from 1 to 2 (Also, we've just realized that our network.host setting is probably not right. Our nodes are co-ordinating over a public network segment and not over a private one as we intended. We fixed that too) Nodes no longer stall but we're still observing the occasional flurry of EsRejectedExecutionExceptions example [2018-02-16T12:54:27,664][DEBUG][o.e.a.s.TransportSearchAction] [redacted\_host2] [9947854] Failed to execute fetch phase org.elasticsearch.transport.RemoteTransportException: [redacted\_host1][192.168.194.50:9300][indices:data/read/search[phase/fetch/id]] Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TcpTransport$RequestHandler@300401b7 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6d6bf90a[Running, pool size = 121, active threads = 121, queued tasks = 1000, completed tasks = 14100175]] This lasts for a few mins and then subsides as if nothing happened Any ideas? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 16, 2018, 11:42am UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/8 "2018-02-16T11:42:04Z")
How many indices/shards do you have in the cluster? ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 16, 2018, 11:58am UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/9 "2018-02-16T11:58:36Z")
6 open indices (excluding system indices). 5 primary shards per index (Only 3 indices are actively indexed and searched on. The rest is backups/rollovers) The biggest and busiest index (by a long way) has 1.6M docs. Search Rate is about 400/s ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 16, 2018, 1:02pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/10 "2018-02-16T13:02:56Z")
What is the size of the indices/shards? ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 16, 2018, 1:13pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/11 "2018-02-16T13:13:26Z")
largest index is 2.8GB. The unused (no searches, no indexing) stale copy of that index is 2.7GB The rest of the indices collectively less than 20MB ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 16, 2018, 1:19pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/12 "2018-02-16T13:19:52Z")
2.8GB is not very large for a shard, so you may want to bring the number of primary shards down to 1 for the indices using the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-shrink-index.html). This will ease the pressure on the queues and may affect latencies (positive or negative), so it may be worth benchmarking it first... ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 16, 2018, 1:56pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/13 "2018-02-16T13:56:34Z")
ok. will try that thanks can it also help to add more nodes to the cluster? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 16, 2018, 2:12pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/14 "2018-02-16T14:12:53Z")
Yes, that could help too. ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 16, 2018, 2:45pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/15 "2018-02-16T14:45:13Z")
I've reduced the num shards from 5 to 1 Will let you know how this goes Have a great weekend and thanks for all the assistance! Amazing ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 17, 2018, 8:27am UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/16 "2018-02-17T08:27:32Z")
Since reducing shards from 5 to 1 we're seeing new behavior Just 20 mins ago, one of the nodes appeared to stall again. Search and GET requests suddenly taking a long time to run. Our client (official elasticsearch js client) times out requests after 5 seconds. Our monitoring system picked up hundreds of timeouts This time around, no EsRejectedExecutionException entries in logs Elasticsearch logs set to DEBUG level. No WARN or ERROR entries for the duration of the stall. Just these... [2018-02-17T09:50:05,287][DEBUG][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [filters.occasion.option\_value.raw][13] took [240.2micros] [2018-02-17T09:50:05,287][DEBUG][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [filters.lense\_type.agg\_key][6] took [184.6micros] [2018-02-17T09:50:05,287][DEBUG][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [filters.length\_detail-bottoms.agg\_key][6] took [165.3micros] [2018-02-17T09:50:05,287][DEBUG][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [filters.leg\_length.option\_value.raw][3] took [111.5micros] [2018-02-17T09:50:05,287][DEBUG][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [filters.fit.agg\_key][21] took [237.8micros] [2018-02-17T09:50:05,287][DEBUG][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [filters.style.agg\_key][565] took [1.6ms] [2018-02-17T09:50:05,287][DEBUG][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [filters.fabric.option\_value.raw][4] took [149micros] [2018-02-17T09:50:05,287][DEBUG][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [filters.strap\_material.agg\_key][9] took [237.6micros] [2018-02-17T09:50:05,287][DEBUG][o.e.i.f.p.SortedSetDVOrdinalsIndexFieldData] global-ordinals [filters.size.agg\_key][3] took [193.6micros] etc. Search rate across the cluster was 40-50/s. Relatively low traffic (for us) ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 17, 2018, 5:47pm UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/17 "2018-02-17T17:47:17Z")
One more worthy observation, this time around its the REPLICA node that is acting up. So far its two occasions in a row. The MASTER node seems to be more stable ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 18, 2018, 8:24am UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/18 "2018-02-18T08:24:44Z")
Just an update. We've reduced heap size from 16G to 4G and seeing a significant reduction in extreme p99 latencies We did that progressively ie. 16, 12, 10, 8, 6, 4 ---
**Author:** ![quintonparker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quintonparker/32/27723_2.png) [@quintonparker](https://discuss.elastic.co/u/quintonparker)\ **Post date:** [February 22, 2018, 8:18am UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/19 "2018-02-22T08:18:47Z")
Update. Reverted heap size to 8GB as we regularly perform large scan & scroll queries which needs a bit more heap else we run into OLD GC runs ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 22, 2018, 8:19am UTC](https://discuss.elastic.co/t/5-5-3-node-stalls-for-no-apparent-reason/119801/20 "2018-03-22T08:19:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.