Parent circuit breaking is always triggered in ES 7.3.2

[Siva_Karan]

Hi Team,

we are using Elasticsearch version 7.3.2

we are getting parent circuit breaking Exception frequently,and it occupies all estimate limit, without bouncing the node how can we rectify it.

Any command or parameters to cleanup the estimated size.

How to cleanup the parent estimate size

   "parent": {
      "limit_size_in_bytes": 11779938713,
      "limit_size": "10.9gb",
      "estimated_size_in_bytes": 9773533832,
      "estimated_size": "9.1gb",
      "overhead": 1.0,
      "tripped": 0
    }

[HenningAndersen]

Hi @Siva_Karan,

which GC are you using? Other users have reported similar issues with G1 GC, fixed by https://github.com/elastic/elasticsearch/pull/46169. If you are running G1 GC, you should copy the changes from that PR into your jvm.options file.

[Siva_Karan]

Hi @HenningAndersen

Thanks for the update.
I'm very new to elastic search 7.3.2 .These are parameters we used for our Elastic search cluster.

1 14 Oct19 ? 1-00:28:52 /ro1/es/jdk1.8.0_131/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-6606390784663210662 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:HeapDumpPath=/ro1/es/logs/cluster-name/node-name/node-name-heapdump.hprof -Xloggc:/ro1/es/logs/cluster-name/node-name/node-name/gc.log -XX:+UseConcMarkSweepGC -Xms15360m -Xmx15360m -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:NewSize=3700m -XX:MaxNewSize=3700m -XX:MaxTenuringThreshold=6 -XX:+PrintTenuringDistribution -XX:SurvivorRatio=6 -XX:+CMSClassUnloadingEnabled -Dio.netty.allocator.type=pooled -XX:MaxDirectMemorySize=8053063680 -Des.path.home=/ro1/es/logs/cluster-name/node-name/node-name -Des.path.conf=/ro1/es/logs/cluster-name/node-name/node-name/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /ro1/es/logs/cluster-name/node-name/node-name/lib/* org.elasticsearch.bootstrap.Elasticsearch -d

[HenningAndersen]

Hi @Siva_Karan,

you are using CMS, so not G1.

I am not sure of where all the options on the command line come from, did you add some of these into jvm.options? Maybe you can share the contents of your jvm.options file?

Also, it would be nice to see the full exception message for this.

It looks like you use a heap of 15360m. The above limit suggests your parent circuit breaker limit might have been lowered to 75%? Could be good to see all your "indices.breaker.*" settings.

Normally, the real memory circuit breaker is in use, which uses current heap size as the estimate. Since 9.1GB is below the CMSInitiatingOccupancyFraction, that looks fine as is. GC will clear some of this when it runs.

However, using real memory circuit breaker with the breaker limit set to around 75% together with CMSInitiatingOccupancyFraction=70 leaves only 5 percent headroom. Since CMS runs concurrently with the application, this is very likely to lead to frequent circuit breaker issues.

I hope the info requested above can help clarify the setup, my comments is my best guess at this time.