# Long GC leading to Missing Shards 0.90.Beta1 **URL:** \ **Category:** Elasticsearch\ **Created:** [March 11, 2013, 11:37pm UTC](https://discuss.elastic.co/t/long-gc-leading-to-missing-shards-0-90-beta1/11106 "2013-03-11T23:37:37Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Greg\_Brown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/greg_brown/32/1373_2.png) [@Greg\_Brown](https://discuss.elastic.co/u/Greg_Brown)\ **Post date:** [March 11, 2013, 11:37pm UTC](https://discuss.elastic.co/t/long-gc-leading-to-missing-shards-0-90-beta1/11106/1 "2013-03-11T23:37:37Z")
Hi, I'm testing an 8 node cluster with 0.90.0.Beta1 under a load (6-7m queries/hr, 300k index ops/hr). Cluster has 40 shards (10 indices) with num\_replicas=2. Had it running continuously for about 5 days then ran into the following sequence: - 05:02: Long (2.3 minutes) GC cycle on one node (es3) seemed to cause loss of connection to the masters resulting in it electing itself. - 05:05: Shards on es3 initially got reallocated on other nodes, and es4 almost immediately started reporting that it was missing a number of shards. - es4 continued to be missing shards for 3 hours - 07:26: Then es3 started missing shards - 09:00: es4 stops reporting missing shards - es3 continues reporting missing shards until we restart the node 5 hours later - A couple of other long GC events 2-3 minutes occurred on es2, es3, and es4 after that first one which makes me think the reallocation of shards is causing them. elasticsearch.yml config that may be relevant: node.max\_local\_storage\_nodes: 1 bootstrap.mlockall: true gateway.recover\_after\_nodes: 5 gateway.recover\_after\_time: 5m gateway.expected\_nodes: 8 discovery.zen.minimum\_master\_nodes: 5 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [list of hosts] threadpool: search: type: fixed index: type: fixed bulk: type: fixed refresh: type: cached monitor.jvm.gc.ParNew.warn: 1000ms monitor.jvm.gc.ParNew.info: 700ms monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s monitor.jvm.gc.ConcurrentMarkSweep.info: 5s A couple of questions: - Could this be an issue with 0.90's new shard allocation? - Any better config settings to avoid getting into such a stuck state in the future? - Any suggestions on avoiding that initial GC cycle? I'm logging GC events, and this first one, and some subsequent ones We have 24 GB allocated and another 20+ GB reserved for the OS. The queries we were running are all more\_like\_this queries with just a short phrase automatically taken from some text. One known issue is we are generating many Exceptions on the cluster right now because the client is sometimes sending null as the "like\_text" parameter. I would hope that periodic exceptions wouldn't cause this problem, but maybe not. Log files are available at: [http://dl.dropbox.com/u/56839351/2013-03-11.global.tar.gz](http://dl.dropbox.com/u/56839351/2013-03-11.global.tar.gz) (sorry for the mess of search exceptions) Thanks for any help -Greg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Greg\_Brown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/greg_brown/32/1373_2.png) [@Greg\_Brown](https://discuss.elastic.co/u/Greg_Brown)\ **Post date:** [March 12, 2013, 12:13am UTC](https://discuss.elastic.co/t/long-gc-leading-to-missing-shards-0-90-beta1/11106/2 "2013-03-12T00:13:37Z")
One other thing I'm investigating is whether the long GC cycles are just a symptom of the shards being reallocated and if a network event caused es3 to momentarily drop out. In which case the question becomes how to properly throttle shard reallocations so they don't cause large GC events. On Monday, March 11, 2013 5:37:37 PM UTC-6, Greg Brown wrote: > Hi, > > I'm testing an 8 node cluster with 0.90.0.Beta1 under a load (6-7m > queries/hr, 300k index ops/hr). Cluster has 40 shards (10 indices) with > num\_replicas=2. Had it running continuously for about 5 days then ran into > the following sequence: > > - 05:02: Long (2.3 minutes) GC cycle on one node (es3) seemed to cause > loss of connection to the masters resulting in it electing itself. > - 05:05: Shards on es3 initially got reallocated on other nodes, and es4 > almost immediately started reporting that it was missing a number of shards. > - es4 continued to be missing shards for 3 hours > - 07:26: Then es3 started missing shards > - 09:00: es4 stops reporting missing shards > - es3 continues reporting missing shards until we restart the node 5 hours > later > - A couple of other long GC events 2-3 minutes occurred on es2, es3, and > es4 after that first one which makes me think the reallocation of shards is > causing them. > > elasticsearch.yml config that may be relevant: > node.max\_local\_storage\_nodes: 1 > bootstrap.mlockall: true > gateway.recover\_after\_nodes: 5 > gateway.recover\_after\_time: 5m > gateway.expected\_nodes: 8 > discovery.zen.minimum\_master\_nodes: 5 > discovery.zen.ping.multicast.enabled: false > discovery.zen.ping.unicast.hosts: [list of hosts] > threadpool: > search: > type: fixed > index: > type: fixed > bulk: > type: fixed > refresh: > type: cached > monitor.jvm.gc.ParNew.warn: 1000ms > monitor.jvm.gc.ParNew.info: 700ms > monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s > monitor.jvm.gc.ConcurrentMarkSweep.info: 5s > > A couple of questions: > > - Could this be an issue with 0.90's new shard allocation? > - Any better config settings to avoid getting into such a stuck state in > the future? > - Any suggestions on avoiding that initial GC cycle? I'm logging GC > events, and this first one, and some subsequent ones We have 24 GB > allocated and another 20+ GB reserved for the OS. The queries we were > running are all more\_like\_this queries with just a short phrase > automatically taken from some text. One known issue is we are generating > many Exceptions on the cluster right now because the client is sometimes > sending null as the "like\_text" parameter. I would hope that periodic > exceptions wouldn't cause this problem, but maybe not. > > Log files are available at: > [http://dl.dropbox.com/u/56839351/2013-03-11.global.tar.gz](http://dl.dropbox.com/u/56839351/2013-03-11.global.tar.gz) (sorry for the > mess of search exceptions) > > Thanks for any help > -Greg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:47am UTC](https://discuss.elastic.co/t/long-gc-leading-to-missing-shards-0-90-beta1/11106/3 "2017-07-06T02:47:04Z")