# Jvm memory problem (Data too large, data for.....) **URL:** \ **Category:** Elasticsearch\ **Created:** [October 21, 2020, 2:07am UTC](https://discuss.elastic.co/t/jvm-memory-problem-data-too-large-data-for/252756 "2020-10-21T02:07:21Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Ariel\_Zach](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Ariel\_Zach](https://discuss.elastic.co/u/Ariel_Zach)\ **Post date:** [October 21, 2020, 2:07am UTC](https://discuss.elastic.co/t/jvm-memory-problem-data-too-large-data-for/252756/1 "2020-10-21T02:07:21Z")
Hello, I am having a problem with the memory of the jvm, I have two clusters with the same configuration and hardware to which I can derive traffic, one with 6.8 and the other with 7.8, the 6.8 cluster has no problem, it has been working well for a long time, but when I pass traffic to 7.8 it throws me the following and the container restarts: ``` [o.e.i.b.HierarchyCircuitBreakerService] [parent] Data too large, data for [indices:data/write/bulk[s]] would be [44058331370/41gb], which is larger than the limit of [44021737062/40.9gb], real usage: [44058313672/41gb], new bytes reserved: [17698/17.2kb], usages [request=32944/32.1kb, fielddata=0/0b, in_flight_requests=2524274/2.4mb, accounting=220509056/210.2mb] ``` I have tried raising the heap to 48gb, but the problem persists and I suspect that no matter how much the heap increases, the same thing will continue to happen. Another thing I tried was to use the same circuit breaker mechanism that 6.8 has, but the same thing happened. Could someone guide me why cluster 6.8 works fine and 7.8 doesn't? Thank you very much. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 21, 2020, 7:01am UTC](https://discuss.elastic.co/t/jvm-memory-problem-data-too-large-data-for/252756/2 "2020-10-21T07:01:27Z")
What is the configuration of the different clusters? What version of the JVM are they using? What does the jvm.options file look like for the two clusters? ---
**Author:** ![Ariel\_Zach](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Ariel\_Zach](https://discuss.elastic.co/u/Ariel_Zach)\ **Post date:** [October 21, 2020, 12:31pm UTC](https://discuss.elastic.co/t/jvm-memory-problem-data-too-large-data-for/252756/3 "2020-10-21T12:31:00Z")
Hello Christian\_Dahlqvist, JVM `ES 6.8.4: 13, ES 7.8.0: 14.0.1` jvm.options: `-Xms31562m -Xmx31562m` Config: ``` # Cluster cluster.name: cluster_name cluster.routing.allocation.awareness.attributes: availability_zone # Node node.name: node_name node.attr.availability_zone: ZONE # Network network.bind_host: 0.0.0.0 network.publish_host: PUBLISH_HOST http.max_content_length: 50mb # Discovery discovery.seed_hosts: master_hosts # Gateway gateway.recover_after_time: 5m gateway.expected_data_nodes: 4 gateway.recover_after_data_nodes: 3 # Various action.destructive_requires_name: true action.auto_create_index: false indices.queries.cache.size: 30% thread_pool.search.queue_size: 3000 # Disable xpack watcher xpack.watcher.enabled: false # Disable xpack ml xpack.ml.enabled: false ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 21, 2020, 12:38pm UTC](https://discuss.elastic.co/t/jvm-memory-problem-data-too-large-data-for/252756/4 "2020-10-21T12:38:22Z")
Can you please share the full jvm options files? That heap size also looks quite large. Can you make sure you are benefitting from compressed pointers (think this should be logged on startup)? ---
**Author:** ![Ariel\_Zach](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Ariel\_Zach](https://discuss.elastic.co/u/Ariel_Zach)\ **Post date:** [October 21, 2020, 12:51pm UTC](https://discuss.elastic.co/t/jvm-memory-problem-data-too-large-data-for/252756/5 "2020-10-21T12:51:38Z")
`/usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Des.cgroups.hierarchy.override=/ -Xms31562m -Xmx31562m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8888 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djdk.security.allowNonCaAnchor=true -Djdk.security.allowNonCaAnchor=true -XX:MaxDirectMemorySize=16550723584 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 21, 2020, 12:53pm UTC](https://discuss.elastic.co/t/jvm-memory-problem-data-too-large-data-for/252756/6 "2020-10-21T12:53:50Z")
Can you please share the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/cluster-stats.html)? ---
**Author:** ![Ariel\_Zach](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Ariel\_Zach](https://discuss.elastic.co/u/Ariel_Zach)\ **Post date:** [October 21, 2020, 12:58pm UTC](https://discuss.elastic.co/t/jvm-memory-problem-data-too-large-data-for/252756/7 "2020-10-21T12:58:21Z")
``` { "_nodes": { "total": 23, "successful": 23, "failed": 0 }, "cluster_name": "xxxxx", "cluster_uuid": "xxxxx", "timestamp": 1603284935214, "status": "green", "indices": { "count": 1, "shards": { "total": 96, "primaries": 32, "replication": 2, "index": { "shards": { "min": 96, "max": 96, "avg": 96 }, "primaries": { "min": 32, "max": 32, "avg": 32 }, "replication": { "min": 2, "max": 2, "avg": 2 } } }, "docs": { "count": 15647623466, "deleted": 7583121974 }, "store": { "size_in_bytes": 19451809656917 }, "fielddata": { "memory_size_in_bytes": 0, "evictions": 0 }, "query_cache": { "memory_size_in_bytes": 0, "total_count": 0, "hit_count": 0, "miss_count": 0, "cache_size": 0, "cache_count": 0, "evictions": 0 }, "completion": { "size_in_bytes": 0 }, "segments": { "count": 6595, "memory_in_bytes": 3335459188, "terms_memory_in_bytes": 323940928, "stored_fields_memory_in_bytes": 2892553904, "term_vectors_memory_in_bytes": 0, "norms_memory_in_bytes": 42227328, "points_memory_in_bytes": 0, "doc_values_memory_in_bytes": 76737028, "index_writer_memory_in_bytes": 1900436482, "version_map_memory_in_bytes": 836704, "fixed_bit_set_memory_in_bytes": 17427776464, "max_unsafe_auto_id_timestamp": -1, "file_sizes": {} }, "mappings": { "field_types": [ { "name": "boolean", "count": 6, "index_count": 1 }, { "name": "date", "count": 8, "index_count": 1 }, { "name": "float", "count": 12, "index_count": 1 }, { "name": "keyword", "count": 105, "index_count": 1 }, { "name": "long", "count": 13, "index_count": 1 }, { "name": "nested", "count": 6, "index_count": 1 }, { "name": "object", "count": 17, "index_count": 1 }, { "name": "text", "count": 105, "index_count": 1 } ] }, "analysis": { "char_filter_types": [], "tokenizer_types": [], "filter_types": [], "analyzer_types": [], "built_in_char_filters": [], "built_in_tokenizers": [], "built_in_filters": [], "built_in_analyzers": [] } }, "nodes": { "count": { "total": 23, "coordinating_only": 0, "data": 20, "ingest": 0, "master": 3, "ml": 0, "remote_cluster_client": 0, "transform": 0, "voting_only": 0 }, "versions": [ "7.8.0" ], "os": { "available_processors": 172, "allocated_processors": 172, "names": [ { "name": "Linux", "count": 23 } ], "pretty_names": [ { "pretty_name": "CentOS Linux 7 (Core)", "count": 23 } ], "mem": { "total_in_bytes": 1358378287104, "free_in_bytes": 47333359616, "used_in_bytes": 1311044927488, "free_percent": 3, "used_percent": 97 } }, "process": { "cpu": { "percent": 406 }, "open_file_descriptors": { "min": 813, "max": 1017, "avg": 967 } }, "jvm": { "max_uptime_in_millis": 515903593, "versions": [ { "version": "14.0.1", "vm_name": "OpenJDK 64-Bit Server VM", "vm_version": "14.0.1+7", "vm_vendor": "AdoptOpenJDK", "bundled_jdk": true, "using_bundled_jdk": true, "count": 23 } ], "mem": { "heap_used_in_bytes": 330711911672, "heap_max_in_bytes": 675515727872 }, "threads": 1986 }, "fs": { "total_in_bytes": 99214998319104, "free_in_bytes": 79270134607872, "available_in_bytes": 74269494890496 }, "network_types": { "transport_types": { "security4": 23 }, "http_types": { "security4": 23 } }, "discovery_types": { "zen": 23 }, "packaging_types": [ { "flavor": "default", "type": "docker", "count": 23 } ], "ingest": { "number_of_pipelines": 2, "processor_stats": { "gsub": { "count": 0, "failed": 0, "current": 0, "time_in_millis": 0 }, "script": { "count": 0, "failed": 0, "current": 0, "time_in_millis": 0 } } } } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 18, 2020, 12:58pm UTC](https://discuss.elastic.co/t/jvm-memory-problem-data-too-large-data-for/252756/8 "2020-11-18T12:58:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.