CircuitBreakingException: [parent] Data too large

Elastic Stack Elasticsearch
1

we upgraded a cluster from 6.8.8 to 7.8.1. ever since this, we are having elastic nodes die with the below error. everything was working fine before the upgrade. i've googled and haven't found anything to really point me in the direction to find what is causing this. any help would be appreciated.

Caused by: org.elasticsearch.transport.RemoteTransportException: [p-gil1-esdata-9131][10.126.57.31:9300][indices:data/write/bulk[s]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [indices:data/write/bulk[s]] would be [31296315898/29.1gb], which is larger than the limit of [30601641984/28.5gb], real usage: [31296311464/29.1gb], new bytes reserved: [4434/4.3kb], usages [request=437866648/417.5mb, fielddata=38637/37.7kb, in_flight_requests=4434/4.3kb, accounting=305567016/291.4mb]
        at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:347) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128) ~[elasticsearch-7.8.0.jar:7.8.0]
2

here is my jvm settings.

## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms30g
-Xmx30g


################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
#-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
3

some of the cluster settings

{
  "defaults" : {
    "indices.analysis.hunspell.dictionary.ignore_case" : "false",
    "indices.analysis.hunspell.dictionary.lazy" : "false",
    "indices.breaker.accounting.limit" : "100%",
    "indices.breaker.accounting.overhead" : "1.0",
    "indices.breaker.fielddata.limit" : "40%",
    "indices.breaker.fielddata.overhead" : "1.03",
    "indices.breaker.fielddata.type" : "memory",
    "indices.breaker.request.limit" : "60%",
    "indices.breaker.request.overhead" : "1.0",
    "indices.breaker.request.type" : "memory",
    "indices.breaker.total.limit" : "95%",
    "indices.breaker.total.use_real_memory" : "true",
    "indices.breaker.type" : "hierarchy",
    "indices.cache.cleanup_interval" : "1m",
    "indices.fielddata.cache.size" : "-1b",
    "indices.id_field_data.enabled" : "true",
    "indices.lifecycle.history_index_enabled" : "true",
    "indices.lifecycle.poll_interval" : "10m",
    "indices.lifecycle.step.master_timeout" : "30s",
    "indices.mapping.dynamic_timeout" : "30s",
    "indices.mapping.max_in_flight_updates" : "10",
    "indices.memory.index_buffer_size" : "10%",
    "indices.memory.interval" : "5s",
    "indices.memory.max_index_buffer_size" : "-1",
    "indices.memory.min_index_buffer_size" : "48mb",
    "indices.memory.shard_inactive_time" : "5m",
    "indices.queries.cache.all_segments" : "false",
    "indices.queries.cache.count" : "10000",
    "indices.queries.cache.size" : "10%",
    "indices.query.bool.max_clause_count" : "1024",
    "indices.query.query_string.allowLeadingWildcard" : "true",
    "indices.query.query_string.analyze_wildcard" : "false",
    "indices.recovery.internal_action_long_timeout" : "1800000ms",
    "indices.recovery.internal_action_timeout" : "15m",
    "indices.recovery.max_concurrent_file_chunks" : "2",
    "indices.recovery.recovery_activity_timeout" : "1800000ms",
    "indices.recovery.retry_delay_network" : "5s",
    "indices.recovery.retry_delay_state_sync" : "500ms",
    "indices.requests.cache.expire" : "0ms",
    "indices.requests.cache.size" : "1%",
    "indices.store.delete.shard.timeout" : "30s"
  }
}
4

example of the breaker settings for one of the many nodes ..

      "breakers" : {
        "request" : {
          "limit_size_in_bytes" : 19327352832,
          "limit_size" : "18gb",
          "estimated_size_in_bytes" : 0,
          "estimated_size" : "0b",
          "overhead" : 1.0,
          "tripped" : 0
        },
        "fielddata" : {
          "limit_size_in_bytes" : 12884901888,
          "limit_size" : "12gb",
          "estimated_size_in_bytes" : 33568,
          "estimated_size" : "32.7kb",
          "overhead" : 1.03,
          "tripped" : 0
        },
        "in_flight_requests" : {
          "limit_size_in_bytes" : 32212254720,
          "limit_size" : "30gb",
          "estimated_size_in_bytes" : 78138,
          "estimated_size" : "76.3kb",
          "overhead" : 2.0,
          "tripped" : 0
        },
        "accounting" : {
          "limit_size_in_bytes" : 32212254720,
          "limit_size" : "30gb",
          "estimated_size_in_bytes" : 257103696,
          "estimated_size" : "245.1mb",
          "overhead" : 1.0,
          "tripped" : 0
        },
        "parent" : {
          "limit_size_in_bytes" : 30601641984,
          "limit_size" : "28.5gb",
          "estimated_size_in_bytes" : 13414933856,
          "estimated_size" : "12.4gb",
          "overhead" : 1.0,
          "tripped" : 0
        }
5

Here is a bigger piece of the output ...

Caused by: org.elasticsearch.transport.RemoteTransportException: [p-gil1-esdata-9131-node1][10.126.57.31:9300][indices:data/write/bulk[s]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [indices:data/write/bulk[s]] would be [31416643296/29.2gb], which is larger than the limit of [30601641984/28.5gb], real usage: [31416626928/29.2gb], new bytes reserved: [16368/15.9kb], usages [request=437866648/417.5mb, fielddata=38637/37.7kb, in_flight_requests=72865140/69.4mb, accounting=305943588/291.7mb]
        at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:347) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.InboundAggregator.checkBreaker(InboundAggregator.java:210) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.InboundAggregator.finishAggregation(InboundAggregator.java:119) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:140) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:73) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) ~[?:?]
[2021-09-21T10:59:03,076][WARN ][o.e.x.m.e.l.LocalExporter] [p-gil1-esdata-9144-node1] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: RemoteTransportException[[p-gil1-esdata-9131-node1][10.126.57.31:9300][indices:data/write/bulk[s]]]; nested: CircuitBreakingException[[parent] Data too large, data for [indices:data/write/bulk[s]] would be [31416643296/29.2gb], which is larger than the limit of [30601641984/28.5gb], real usage: [31416626928/29.2gb], new bytes reserved: [16368/15.9kb], usages [request=437866648/417.5mb, fielddata=38637/37.7kb, in_flight_requests=72865140/69.4mb, accounting=305943588/291.7mb]];
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-7.8.0.jar:7.8.0]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:126) [x-pack-monitoring-7.8.0.jar:7.8.0]
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:108) [x-pack-monitoring-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:163) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:555) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:550) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:98) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:794) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:752) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:639) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1173) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:233) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:225) [elasticsearch-7.8.0.jar:7.8.0
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.