Jvm memory problem (Data too large, data for.....)

Elastic Stack Elasticsearch
1

Hello,

I am having a problem with the memory of the jvm, I have two clusters with the same configuration and hardware to which I can derive traffic, one with 6.8 and the other with 7.8, the 6.8 cluster has no problem, it has been working well for a long time, but when I pass traffic to 7.8 it throws me the following and the container restarts:

[o.e.i.b.HierarchyCircuitBreakerService] [parent] Data too large, data for [indices:data/write/bulk[s]] would be [44058331370/41gb], which is larger than the limit of [44021737062/40.9gb], real usage: [44058313672/41gb], new bytes reserved: [17698/17.2kb], usages [request=32944/32.1kb, fielddata=0/0b, in_flight_requests=2524274/2.4mb, accounting=220509056/210.2mb]

I have tried raising the heap to 48gb, but the problem persists and I suspect that no matter how much the heap increases, the same thing will continue to happen.

Another thing I tried was to use the same circuit breaker mechanism that 6.8 has, but the same thing happened.

Could someone guide me why cluster 6.8 works fine and 7.8 doesn't?

Thank you very much.

2

What is the configuration of the different clusters? What version of the JVM are they using? What does the jvm.options file look like for the two clusters?

3

Hello Christian_Dahlqvist,

JVM
ES 6.8.4: 13, ES 7.8.0: 14.0.1

jvm.options:

-Xms31562m -Xmx31562m

Config:

# Cluster
cluster.name: cluster_name

cluster.routing.allocation.awareness.attributes: availability_zone

# Node
node.name: node_name

node.attr.availability_zone: ZONE

# Network
network.bind_host: 0.0.0.0
network.publish_host: PUBLISH_HOST
http.max_content_length: 50mb

# Discovery
discovery.seed_hosts: master_hosts

# Gateway
gateway.recover_after_time: 5m
gateway.expected_data_nodes: 4
gateway.recover_after_data_nodes: 3

# Various
action.destructive_requires_name: true
action.auto_create_index: false

indices.queries.cache.size: 30%

thread_pool.search.queue_size: 3000

# Disable xpack watcher
xpack.watcher.enabled: false

# Disable xpack ml
xpack.ml.enabled: false
4

Can you please share the full jvm options files? That heap size also looks quite large. Can you make sure you are benefitting from compressed pointers (think this should be logged on startup)?

5

/usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Des.cgroups.hierarchy.override=/ -Xms31562m -Xmx31562m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8888 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djdk.security.allowNonCaAnchor=true -Djdk.security.allowNonCaAnchor=true -XX:MaxDirectMemorySize=16550723584 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch

6

Can you please share the full output of the cluster stats API?

7 
{
  "_nodes": {
    "total": 23,
    "successful": 23,
    "failed": 0
  },
  "cluster_name": "xxxxx",
  "cluster_uuid": "xxxxx",
  "timestamp": 1603284935214,
  "status": "green",
  "indices": {
    "count": 1,
    "shards": {
      "total": 96,
      "primaries": 32,
      "replication": 2,
      "index": {
        "shards": {
          "min": 96,
          "max": 96,
          "avg": 96
        },
        "primaries": {
          "min": 32,
          "max": 32,
          "avg": 32
        },
        "replication": {
          "min": 2,
          "max": 2,
          "avg": 2
        }
      }
    },
    "docs": {
      "count": 15647623466,
      "deleted": 7583121974
    },
    "store": {
      "size_in_bytes": 19451809656917
    },
    "fielddata": {
      "memory_size_in_bytes": 0,
      "evictions": 0
    },
    "query_cache": {
      "memory_size_in_bytes": 0,
      "total_count": 0,
      "hit_count": 0,
      "miss_count": 0,
      "cache_size": 0,
      "cache_count": 0,
      "evictions": 0
    },
    "completion": {
      "size_in_bytes": 0
    },
    "segments": {
      "count": 6595,
      "memory_in_bytes": 3335459188,
      "terms_memory_in_bytes": 323940928,
      "stored_fields_memory_in_bytes": 2892553904,
      "term_vectors_memory_in_bytes": 0,
      "norms_memory_in_bytes": 42227328,
      "points_memory_in_bytes": 0,
      "doc_values_memory_in_bytes": 76737028,
      "index_writer_memory_in_bytes": 1900436482,
      "version_map_memory_in_bytes": 836704,
      "fixed_bit_set_memory_in_bytes": 17427776464,
      "max_unsafe_auto_id_timestamp": -1,
      "file_sizes": {}
    },
    "mappings": {
      "field_types": [
        {
          "name": "boolean",
          "count": 6,
          "index_count": 1
        },
        {
          "name": "date",
          "count": 8,
          "index_count": 1
        },
        {
          "name": "float",
          "count": 12,
          "index_count": 1
        },
        {
          "name": "keyword",
          "count": 105,
          "index_count": 1
        },
        {
          "name": "long",
          "count": 13,
          "index_count": 1
        },
        {
          "name": "nested",
          "count": 6,
          "index_count": 1
        },
        {
          "name": "object",
          "count": 17,
          "index_count": 1
        },
        {
          "name": "text",
          "count": 105,
          "index_count": 1
        }
      ]
    },
    "analysis": {
      "char_filter_types": [],
      "tokenizer_types": [],
      "filter_types": [],
      "analyzer_types": [],
      "built_in_char_filters": [],
      "built_in_tokenizers": [],
      "built_in_filters": [],
      "built_in_analyzers": []
    }
  },
  "nodes": {
    "count": {
      "total": 23,
      "coordinating_only": 0,
      "data": 20,
      "ingest": 0,
      "master": 3,
      "ml": 0,
      "remote_cluster_client": 0,
      "transform": 0,
      "voting_only": 0
    },
    "versions": [
      "7.8.0"
    ],
    "os": {
      "available_processors": 172,
      "allocated_processors": 172,
      "names": [
        {
          "name": "Linux",
          "count": 23
        }
      ],
      "pretty_names": [
        {
          "pretty_name": "CentOS Linux 7 (Core)",
          "count": 23
        }
      ],
      "mem": {
        "total_in_bytes": 1358378287104,
        "free_in_bytes": 47333359616,
        "used_in_bytes": 1311044927488,
        "free_percent": 3,
        "used_percent": 97
      }
    },
    "process": {
      "cpu": {
        "percent": 406
      },
      "open_file_descriptors": {
        "min": 813,
        "max": 1017,
        "avg": 967
      }
    },
    "jvm": {
      "max_uptime_in_millis": 515903593,
      "versions": [
        {
          "version": "14.0.1",
          "vm_name": "OpenJDK 64-Bit Server VM",
          "vm_version": "14.0.1+7",
          "vm_vendor": "AdoptOpenJDK",
          "bundled_jdk": true,
          "using_bundled_jdk": true,
          "count": 23
        }
      ],
      "mem": {
        "heap_used_in_bytes": 330711911672,
        "heap_max_in_bytes": 675515727872
      },
      "threads": 1986
    },
    "fs": {
      "total_in_bytes": 99214998319104,
      "free_in_bytes": 79270134607872,
      "available_in_bytes": 74269494890496
    },
    "network_types": {
      "transport_types": {
        "security4": 23
      },
      "http_types": {
        "security4": 23
      }
    },
    "discovery_types": {
      "zen": 23
    },
    "packaging_types": [
      {
        "flavor": "default",
        "type": "docker",
        "count": 23
      }
    ],
    "ingest": {
      "number_of_pipelines": 2,
      "processor_stats": {
        "gsub": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "script": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        }
      }
    }
  }
}
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.