Shard has exceeded the maximum number of retries

Elastic Stack Elasticsearch
5

Ahh, seem that the actual reason is the unpredictable mem usage rise after upgrade!

That make me remember some discuss, about the jvm setting that trigger the breaker happens

  1. parent-circuit-breaker-calculation-seems-to-be-wrong-with-version-7-x
  2. circuitbreakingexception-parent-data-too-large-in-es-7-x
  3. g1gc-cause-circuitbreakingexception-parent-data-too-large-on-7-1-1

And which also make breaker occasionally happened in our cluster after upgrade to 7.4.

By then we change the jvm default setting to the bellow settings in our tsdb use case:
And breakers (to us are caused by indices.breaker.request) never happen again!

################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms31g
-Xmx31g

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
# # # # # # -XX:+UseConcMarkSweepGC
# # # # # # -XX:CMSInitiatingOccupancyFraction=75
# # # # # # -XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1GC is only supported on JDK version 10 or later.
# To use G1GC uncomment the lines below.
10-:-XX:-UseConcMarkSweepGC
10-:-XX:-UseCMSInitiatingOccupancyOnly
10-:-XX:+UseG1GC
## optimizations
10-:-XX:InitiatingHeapOccupancyPercent=30       ##  <---- main reason, don't set it higher than 40
10-:-XX:MaxGCPauseMillis=5                                   ##  <---- let gc happens more frequently
10-:-XX:GCPauseIntervalMillis=10                           ##  <---- let gc happens more frequently
-XX:+UseCompressedOops
-XX:+DisableExplicitGC

## optimizations

Wish it can fix ur problem!