Hi,
I have upgraded my cluster from 8.11 to 8.17 and since it keeps crashing due to full heap used. I have upped the heap memory but it did not solve the issue. After some time with heap memory use steady, it starts growing up until using all the heap. Only a restart solves for a few days.
I keep getting these logs for failed GC:
[2025-01-08T05:08:11,647][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [0], before [12256000632], after [11686031128]
[2025-01-08T05:08:11,648][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [2], before [12256000632], after [11686031128]
[2025-01-08T05:08:11,649][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [3], before [12256000632], after [11686031128]
[2025-01-08T05:08:11,657][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [11], before [12256000632], after [11686031128]
[2025-01-08T05:08:11,659][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [13], before [12306332280], after [11694419736]
[2025-01-08T05:08:11,661][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [15], before [12323109496], after [11694419736]
[2025-01-08T05:08:11,667][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [21], before [12365052536], after [11694419736]
[2025-01-08T05:08:11,661][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [15], before [12339886712], after [11694419736]
[2025-01-08T05:08:11,663][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [17], before [12356663928], after [11694419736]
[2025-01-08T05:08:12,960][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] attempting to trigger G1GC due to high heap usage [12256456472]
[2025-01-08T05:08:13,019][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] GC did bring memory usage down, before [12256456472], after [12253451072], allocations [68], duration [58]
[2025-01-08T05:08:13,019][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [0], before [12441005848], after [12253451072]
[2025-01-08T05:08:13,031][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] attempt to trigger young GC failed to bring memory down, triggering full GC
[2025-01-08T05:08:18,249][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [13], before [12253451072], after [11553765528]
[2025-01-08T05:08:18,252][WARN ][o.e.t.ThreadPool ] [node-elk3] timer thread slept for [5.3s/5371ms] on absolute clock which is above the warn threshold of [5000ms]
[2025-01-08T05:08:18,252][WARN ][o.e.t.ThreadPool ] [node-elk3] timer thread slept for [5.3s/5371003740ns] on relative clock which is above the warn threshold of [5000ms]
[2025-01-08T05:08:18,257][INFO ][o.e.m.j.JvmGcMonitorService] [node-elk3] [gc][old][182767][52] duration [5.2s], collections [1]/[5.6s], total [5.2s]/[4.3m], memory [11.2gb]->[10.7gb]/[12gb], all_pools {[CodeHeap 'non-nmethods'] [2.7mb]->[2.7mb]/[5.5mb]}{[Metaspace] [208mb]->[208mb]/[0b]}{[CodeHeap 'profiled nmethods'] [7.1mb]->[7.1mb]/[117.2mb]}{[Compressed Class Space] [28.1mb]->[28.1mb]/[1gb]}{[young] [344mb]->[8mb]/[0b]}{[old] [10.8gb]->[10.7gb]/[12gb]}{[survivor] [39.7mb]->[0b]/[0b]}{[CodeHeap 'non-profiled nmethods'] [48mb]->[48mb]/[117.2mb]}
[2025-01-08T05:08:18,257][WARN ][o.e.m.j.JvmGcMonitorService] [node-elk3] [gc][182767] overhead, spent [5.2s] collecting in the last [5.6s]
[2025-01-08T05:53:06,124][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] attempting to trigger G1GC due to high heap usage [12247381632]
[2025-01-08T05:53:06,218][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] attempt to trigger young GC failed to bring memory down, triggering full GC
[2025-01-08T05:53:11,424][WARN ][o.e.t.ThreadPool ] [node-elk3] timer thread slept for [5.3s/5314ms] on absolute clock which is above the warn threshold of [5000ms]
[2025-01-08T05:53:11,424][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] GC did bring memory usage down, before [12247381632], after [11668240352], allocations [67], duration [5300]
[2025-01-08T05:53:11,424][WARN ][o.e.t.ThreadPool ] [node-elk3] timer thread slept for [5.3s/5314173953ns] on relative clock which is above the warn threshold of [5000ms]
[2025-01-08T05:53:11,425][INFO ][o.e.m.j.JvmGcMonitorService] [node-elk3] [gc][old][184943][168] duration [5.1s], collections [1]/[5.9s], total [5.1s]/[13.9m], memory [11.1gb]->[10.8gb]/[12gb], all_pools {[CodeHeap 'non-nmethods'] [2.7mb]->[2.7mb]/[5.5mb]}{[Metaspace] [208.5mb]->[208.5mb]/[0b]}{[CodeHeap 'profiled nmethods'] [7mb]->[7mb]/[117.2mb]}{[Compressed Class Space] [28.2mb]->[28.2mb]/[1gb]}{[young] [304mb]->[0b]/[0b]}{[old] [10.8gb]->[10.8gb]/[12gb]}{[survivor] [19.6mb]->[0b]/[0b]}{[CodeHeap 'non-profiled nmethods'] [48.7mb]->[48.7mb]/[117.2mb]}
[2025-01-08T05:53:11,425][WARN ][o.e.m.j.JvmGcMonitorService] [node-elk3] [gc][184943] overhead, spent [5.2s] collecting in the last [5.9s]
[2025-01-08T05:53:11,425][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] could not acquire lock within 500ms when attempting to trigger G1GC due to high heap usage
[2025-01-08T05:53:11,425][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [0], before [12389987968], after [11668240352]
[2025-01-08T05:53:11,426][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] could not acquire lock within 500ms when attempting to trigger G1GC due to high heap usage
[2025-01-08T05:53:11,426][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [node-elk3] memory usage down after [0], before [12322879104], after [11668240352]
It does not seem to be the same issue than the one described in OOM since 8.16.1 with openjdk23 as I checked
sudo sysctl -a | grep max_map_count
vm.max_map_count = 262144
sudo wc -l /proc/374416/maps
19280 /proc/374416/maps
Here is the jvm.options config:
sudo grep -v '^#' /etc/elasticsearch/jvm.options
-XX:+UseG1GC
-Djava.io.tmpdir=${ES_TMPDIR}
20-:--add-modules=jdk.incubator.vector
23:-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.setAsTypeCache
23:-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.asTypeUncached
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log
-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m
It was using 7g before the upgrade and was stable, now it is set to 12g on both data nodes, the third master node is not a data node and is stable.
sudo grep -v '^\s*#' /etc/elasticsearch/jvm.options.d/jvm.options
-Xms12g
-Xmx12g
GET _nodes/_all/jvm
{
"_nodes": {
"total": 3,
"successful": 3,
"failed": 0
},
"cluster_name": "redacted",
"nodes": {
"xxxxxxx-xxxxxxx": {
"name": "elk-1",
"transport_address": "[xxxx:xxxx:xxxx:xxxx::xxxx]:9300",
"host": "xxxx:xxxx:xxxx:xxxx::xxxx%eth0",
"ip": "xxxx:xxxx:xxxx:xxxx::xxxx",
"version": "8.17.0",
"transport_version": 8797002,
"index_version": 8521000,
"component_versions": {
"transform_config_version": 10000099,
"ml_config_version": 12000099,
"api_key_version": 8150099
},
"build_flavor": "default",
"build_type": "deb",
"build_hash": "2b6a7fed44faa321997703718f07ee0420804b41",
"roles": [
"data",
"data_cold",
"data_content",
"data_frozen",
"data_hot",
"data_warm",
"ingest",
"master",
"ml",
"remote_cluster_client",
"transform"
],
"attributes": {
"ml.allocated_processors_double": "8.0",
"ml.allocated_processors": "8",
"ml.machine_memory": "25199276032",
"transform.config_version": "10.0.0",
"xpack.installed": "true",
"ml.config_version": "12.0.0",
"ml.max_jvm_size": "12884901888"
},
"jvm": {
"pid": 262863,
"version": "23",
"vm_name": "OpenJDK 64-Bit Server VM",
"vm_version": "23+37-2369",
"vm_vendor": "Oracle Corporation",
"using_bundled_jdk": true,
"start_time_in_millis": 1736146349777,
"mem": {
"heap_init_in_bytes": 12884901888,
"heap_max_in_bytes": 12884901888,
"non_heap_init_in_bytes": 7667712,
"non_heap_max_in_bytes": 0,
"direct_max_in_bytes": 0
},
"gc_collectors": [
"G1 Young Generation",
"G1 Concurrent GC",
"G1 Old Generation"
],
"memory_pools": [
"CodeHeap 'non-nmethods'",
"Metaspace",
"CodeHeap 'profiled nmethods'",
"Compressed Class Space",
"G1 Eden Space",
"G1 Old Gen",
"G1 Survivor Space",
"CodeHeap 'non-profiled nmethods'"
],
"using_compressed_ordinary_object_pointers": "true",
"input_arguments": [
"-Des.networkaddress.cache.ttl=60",
"-Des.networkaddress.cache.negative.ttl=10",
"-XX:+AlwaysPreTouch",
"-Xss1m",
"-Djava.awt.headless=true",
"-Dfile.encoding=UTF-8",
"-Djna.nosys=true",
"-XX:-OmitStackTraceInFastThrow",
"-Dio.netty.noUnsafe=true",
"-Dio.netty.noKeySetOptimization=true",
"-Dio.netty.recycler.maxCapacityPerThread=0",
"-Dlog4j.shutdownHookEnabled=false",
"-Dlog4j2.disable.jmx=true",
"-Dlog4j2.formatMsgNoLookups=true",
"-Djava.locale.providers=CLDR",
"-Des.distribution.type=deb",
"--enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core",
"-XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log",
"-Djava.security.manager=allow",
"-XX:+UseG1GC",
"-Djava.io.tmpdir=/tmp/elasticsearch-1692151378270143613",
"--add-modules=jdk.incubator.vector",
"-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.setAsTypeCache",
"-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.asTypeUncached",
"-XX:+HeapDumpOnOutOfMemoryError",
"-XX:+ExitOnOutOfMemoryError",
"-XX:HeapDumpPath=/var/lib/elasticsearch",
"-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log",
"-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m",
"-Xms12g",
"-Xmx12g",
"-XX:MaxDirectMemorySize=6442450944",
"-XX:InitiatingHeapOccupancyPercent=30",
"-XX:G1ReservePercent=25",
"--module-path=/usr/share/elasticsearch/lib",
"--add-modules=jdk.net",
"--add-modules=ALL-MODULE-PATH",
"-Djdk.module.main=org.elasticsearch.server"
]
}
},
"xxxxxxxx_xxxxxxxx": {
"name": "es2",
"transport_address": "[xxxx:xxxx:xxxx:xxxx::xxxx]:9300",
"host": "xxxx:xxxx:xxxx:xxxx::xxxx%eth0",
"ip": "xxxx:xxxx:xxxx:xxxx::xxxx",
"version": "8.17.0",
"transport_version": 8797002,
"index_version": 8521000,
"component_versions": {
"api_key_version": 8150099,
"transform_config_version": 10000099,
"ml_config_version": 12000099
},
"build_flavor": "default",
"build_type": "deb",
"build_hash": "2b6a7fed44faa321997703718f07ee0420804b41",
"roles": [
"master",
"voting_only"
],
"attributes": {
"transform.config_version": "10.0.0",
"ml.config_version": "12.0.0",
"xpack.installed": "true"
},
"jvm": {
"pid": 1247,
"version": "23",
"vm_name": "OpenJDK 64-Bit Server VM",
"vm_version": "23+37-2369",
"vm_vendor": "Oracle Corporation",
"using_bundled_jdk": true,
"start_time_in_millis": 1734686712271,
"mem": {
"heap_init_in_bytes": 1031798784,
"heap_max_in_bytes": 1031798784,
"non_heap_init_in_bytes": 7667712,
"non_heap_max_in_bytes": 0,
"direct_max_in_bytes": 0
},
"gc_collectors": [
"G1 Young Generation",
"G1 Concurrent GC",
"G1 Old Generation"
],
"memory_pools": [
"CodeHeap 'non-nmethods'",
"Metaspace",
"CodeHeap 'profiled nmethods'",
"Compressed Class Space",
"G1 Eden Space",
"G1 Old Gen",
"G1 Survivor Space",
"CodeHeap 'non-profiled nmethods'"
],
"using_compressed_ordinary_object_pointers": "true",
"input_arguments": [
"-Des.networkaddress.cache.ttl=60",
"-Des.networkaddress.cache.negative.ttl=10",
"-XX:+AlwaysPreTouch",
"-Xss1m",
"-Djava.awt.headless=true",
"-Dfile.encoding=UTF-8",
"-Djna.nosys=true",
"-XX:-OmitStackTraceInFastThrow",
"-Dio.netty.noUnsafe=true",
"-Dio.netty.noKeySetOptimization=true",
"-Dio.netty.recycler.maxCapacityPerThread=0",
"-Dlog4j.shutdownHookEnabled=false",
"-Dlog4j2.disable.jmx=true",
"-Dlog4j2.formatMsgNoLookups=true",
"-Djava.locale.providers=CLDR",
"-Des.distribution.type=deb",
"--enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core",
"-XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log",
"-Djava.security.manager=allow",
"-XX:+UseG1GC",
"-Djava.io.tmpdir=/tmp/elasticsearch-17403265704748004512",
"--add-modules=jdk.incubator.vector",
"-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.setAsTypeCache",
"-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.asTypeUncached",
"-XX:+HeapDumpOnOutOfMemoryError",
"-XX:+ExitOnOutOfMemoryError",
"-XX:HeapDumpPath=/var/lib/elasticsearch",
"-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log",
"-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m",
"-Xms983m",
"-Xmx983m",
"-XX:MaxDirectMemorySize=515899392",
"-XX:G1HeapRegionSize=4m",
"-XX:InitiatingHeapOccupancyPercent=30",
"-XX:G1ReservePercent=15",
"--module-path=/usr/share/elasticsearch/lib",
"--add-modules=jdk.net",
"--add-modules=ALL-MODULE-PATH",
"-Djdk.module.main=org.elasticsearch.server"
]
}
},
"xxxxxxxxx_xxxxxxxx": {
"name": "node-elk3",
"transport_address": "[xxxx:xxxx:xxxx:xxxx::xxxx]:9300",
"host": "xxxx:xxxx:xxxx:xxxx::xxxx%ens18",
"ip": "xxxx:xxxx:xxxx:xxxx::xxxx",
"version": "8.17.0",
"transport_version": 8797002,
"index_version": 8521000,
"component_versions": {
"ml_config_version": 12000099,
"transform_config_version": 10000099,
"api_key_version": 8150099
},
"build_flavor": "default",
"build_type": "deb",
"build_hash": "2b6a7fed44faa321997703718f07ee0420804b41",
"roles": [
"data",
"data_cold",
"data_content",
"data_frozen",
"data_hot",
"data_warm",
"ingest",
"master",
"ml",
"remote_cluster_client",
"transform"
],
"attributes": {
"ml.allocated_processors_double": "8.0",
"ml.allocated_processors": "8",
"ml.machine_memory": "25199243264",
"transform.config_version": "10.0.0",
"xpack.installed": "true",
"ml.config_version": "12.0.0",
"ml.max_jvm_size": "12884901888"
},
"jvm": {
"pid": 468492,
"version": "23",
"vm_name": "OpenJDK 64-Bit Server VM",
"vm_version": "23+37-2369",
"vm_vendor": "Oracle Corporation",
"using_bundled_jdk": true,
"start_time_in_millis": 1736336395708,
"mem": {
"heap_init_in_bytes": 12884901888,
"heap_max_in_bytes": 12884901888,
"non_heap_init_in_bytes": 7667712,
"non_heap_max_in_bytes": 0,
"direct_max_in_bytes": 0
},
"gc_collectors": [
"G1 Young Generation",
"G1 Concurrent GC",
"G1 Old Generation"
],
"memory_pools": [
"CodeHeap 'non-nmethods'",
"Metaspace",
"CodeHeap 'profiled nmethods'",
"Compressed Class Space",
"G1 Eden Space",
"G1 Old Gen",
"G1 Survivor Space",
"CodeHeap 'non-profiled nmethods'"
],
"using_compressed_ordinary_object_pointers": "true",
"input_arguments": [
"-Des.networkaddress.cache.ttl=60",
"-Des.networkaddress.cache.negative.ttl=10",
"-XX:+AlwaysPreTouch",
"-Xss1m",
"-Djava.awt.headless=true",
"-Dfile.encoding=UTF-8",
"-Djna.nosys=true",
"-XX:-OmitStackTraceInFastThrow",
"-Dio.netty.noUnsafe=true",
"-Dio.netty.noKeySetOptimization=true",
"-Dio.netty.recycler.maxCapacityPerThread=0",
"-Dlog4j.shutdownHookEnabled=false",
"-Dlog4j2.disable.jmx=true",
"-Dlog4j2.formatMsgNoLookups=true",
"-Djava.locale.providers=CLDR",
"-Des.distribution.type=deb",
"--enable-native-access=org.elasticsearch.nativeaccess,org.apache.lucene.core",
"-XX:ReplayDataFile=/var/log/elasticsearch/replay_pid%p.log",
"-Djava.security.manager=allow",
"-XX:+UseG1GC",
"-Djava.io.tmpdir=/tmp/elasticsearch-1828706303470085143",
"--add-modules=jdk.incubator.vector",
"-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.setAsTypeCache",
"-XX:CompileCommand=dontinline,java/lang/invoke/MethodHandle.asTypeUncached",
"-XX:+HeapDumpOnOutOfMemoryError",
"-XX:+ExitOnOutOfMemoryError",
"-XX:HeapDumpPath=/var/lib/elasticsearch",
"-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log",
"-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m",
"-Xms12g",
"-Xmx12g",
"-XX:MaxDirectMemorySize=6442450944",
"-XX:InitiatingHeapOccupancyPercent=30",
"-XX:G1ReservePercent=25",
"--module-path=/usr/share/elasticsearch/lib",
"--add-modules=jdk.net",
"--add-modules=ALL-MODULE-PATH",
"-Djdk.module.main=org.elasticsearch.server"
]
}
}
}
}
Servers are on Ubuntu 24.04 and Elasticsearch is using bundled Java
{
"_nodes": {
"total": 3,
"successful": 3,
"failed": 0
},
"cluster_name": "redacted",
"cluster_uuid": "some_uuid",
"timestamp": 1736332710536,
"status": "green",
"indices": {
"count": 695,
"shards": {
"total": 1418,
"primaries": 709,
"replication": 1,
"index": {
"shards": {
"min": 2,
"max": 6,
"avg": 2.0402877697841726
},
"primaries": {
"min": 1,
"max": 3,
"avg": 1.0201438848920863
},
"replication": {
"min": 1,
"max": 1,
"avg": 1
}
}
},
"docs": {
"count": 1152459769,
"deleted": 402324,
"total_size": "354.9gb",
"total_size_in_bytes": 381118982398
},
"store": {
"size": "713.8gb",
"size_in_bytes": 766525582487,
"total_data_set_size": "713.8gb",
"total_data_set_size_in_bytes": 766525582487,
"reserved": "0b",
"reserved_in_bytes": 0
},
"fielddata": {
"memory_size": "414.6kb",
"memory_size_in_bytes": 424552,
"evictions": 905,
"global_ordinals": {
"build_time": "1.7m",
"build_time_in_millis": 103310
}
},
"query_cache": {
"memory_size": "2.3gb",
"memory_size_in_bytes": 2576805750,
"total_count": 10094819,
"hit_count": 3144757,
"miss_count": 6950062,
"cache_size": 145463,
"cache_count": 235321,
"evictions": 89858
},
"completion": {
"size": "0b",
"size_in_bytes": 0
},
"segments": {
"count": 34124,
"memory": "0b",
"memory_in_bytes": 0,
"terms_memory": "0b",
"terms_memory_in_bytes": 0,
"stored_fields_memory": "0b",
"stored_fields_memory_in_bytes": 0,
"term_vectors_memory": "0b",
"term_vectors_memory_in_bytes": 0,
"norms_memory": "0b",
"norms_memory_in_bytes": 0,
"points_memory": "0b",
"points_memory_in_bytes": 0,
"doc_values_memory": "0b",
"doc_values_memory_in_bytes": 0,
"index_writer_memory": "108.5mb",
"index_writer_memory_in_bytes": 113788976,
"version_map_memory": "226.7kb",
"version_map_memory_in_bytes": 232207,
"fixed_bit_set": "7.7mb",
"fixed_bit_set_memory_in_bytes": 8108384,
"max_unsafe_auto_id_timestamp": 1736330338381,
"file_sizes": {}
},
"mappings": {
"total_field_count": 769680,
"total_deduplicated_field_count": 100847,
"total_deduplicated_mapping_size": "725.7kb",
"total_deduplicated_mapping_size_in_bytes": 743159,
"field_types": [
{
"name": "aggregate_metric_double",
"count": 202,
"index_count": 101,
"script_count": 0
},
{
"name": "alias",
"count": 2724,
"index_count": 139,
"script_count": 0
},
{
"name": "binary",
"count": 3,
"index_count": 3,
"script_count": 0
},
{
"name": "boolean",
"count": 15155,
"index_count": 403,
"script_count": 0
},
{
"name": "byte",
"count": 98,
"index_count": 98,
"script_count": 0
},
{
"name": "constant_keyword",
"count": 2142,
"index_count": 490,
"script_count": 0
},
{
"name": "counted_keyword",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "date",
"count": 21153,
"index_count": 642,
"script_count": 0
},
{
"name": "date_nanos",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "date_range",
"count": 13,
"index_count": 13,
"script_count": 0
},
{
"name": "double",
"count": 3479,
"index_count": 132,
"script_count": 0
},
{
"name": "double_range",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "flattened",
"count": 4265,
"index_count": 126,
"script_count": 0
},
{
"name": "float",
"count": 4166,
"index_count": 228,
"script_count": 0
},
{
"name": "float_range",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "geo_point",
"count": 1345,
"index_count": 207,
"script_count": 0
},
{
"name": "geo_shape",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "half_float",
"count": 87,
"index_count": 28,
"script_count": 0
},
{
"name": "histogram",
"count": 109,
"index_count": 109,
"script_count": 0
},
{
"name": "integer",
"count": 584,
"index_count": 79,
"script_count": 0
},
{
"name": "integer_range",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "ip",
"count": 13221,
"index_count": 350,
"script_count": 0
},
{
"name": "ip_range",
"count": 91,
"index_count": 91,
"script_count": 0
},
{
"name": "keyword",
"count": 455180,
"index_count": 645,
"script_count": 1,
"lang": [
"painless"
],
"lines_max": 15,
"lines_total": 15,
"chars_max": 522,
"chars_total": 522,
"source_max": 0,
"source_total": 0,
"doc_max": 0,
"doc_total": 0
},
{
"name": "long",
"count": 112223,
"index_count": 597,
"script_count": 0
},
{
"name": "long_range",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "match_only_text",
"count": 7466,
"index_count": 334,
"script_count": 0
},
{
"name": "nested",
"count": 1801,
"index_count": 164,
"script_count": 0
},
{
"name": "object",
"count": 101675,
"index_count": 641,
"script_count": 0
},
{
"name": "rank_features",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "scaled_float",
"count": 364,
"index_count": 150,
"script_count": 0
},
{
"name": "semantic_text",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "shape",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "short",
"count": 9292,
"index_count": 98,
"script_count": 0
},
{
"name": "text",
"count": 10872,
"index_count": 264,
"script_count": 0
},
{
"name": "unsigned_long",
"count": 96,
"index_count": 14,
"script_count": 0
},
{
"name": "version",
"count": 28,
"index_count": 28,
"script_count": 0
},
{
"name": "wildcard",
"count": 1836,
"index_count": 144,
"script_count": 0
}
],
"runtime_field_types": [],
"source_modes": {
"stored": 647
}
},
"analysis": {
"char_filter_types": [],
"tokenizer_types": [],
"filter_types": [],
"analyzer_types": [],
"built_in_char_filters": [],
"built_in_tokenizers": [],
"built_in_filters": [],
"built_in_analyzers": [],
"synonyms": {}
},
"versions": [
{
"version": "7.6.1",
"index_count": 4,
"primary_shard_count": 4,
"total_primary_size": "2.3mb",
"total_primary_bytes": 2486588
},
{
"version": "7.8.0",
"index_count": 5,
"primary_shard_count": 5,
"total_primary_size": "2.9mb",
"total_primary_bytes": 3119813
},
{
"version": "7.9.2",
"index_count": 1,
"primary_shard_count": 1,
"total_primary_size": "12.5mb",
"total_primary_bytes": 13126699
},
{
"version": "7.10.0",
"index_count": 2,
"primary_shard_count": 2,
"total_primary_size": "12.8mb",
"total_primary_bytes": 13481193
},
{
"version": "7.14.1",
"index_count": 7,
"primary_shard_count": 7,
"total_primary_size": "6.2gb",
"total_primary_bytes": 6727807089
},
{
"version": "7.16.0",
"index_count": 51,
"primary_shard_count": 51,
"total_primary_size": "9.3gb",
"total_primary_bytes": 10068147476
},
{
"version": "7.17.9",
"index_count": 40,
"primary_shard_count": 40,
"total_primary_size": "56.4mb",
"total_primary_bytes": 59185531
},
{
"version": "8.11.0-8.11.4",
"index_count": 313,
"primary_shard_count": 313,
"total_primary_size": "30.3gb",
"total_primary_bytes": 32591243631
},
{
"version": "8.17.0",
"index_count": 272,
"primary_shard_count": 286,
"total_primary_size": "309.7gb",
"total_primary_bytes": 332593709921
}
],
"search": {
"total": 1947871,
"queries": {
"bool": 1858961,
"prefix": 79,
"match": 121870,
"range": 1703200,
"nested": 3086,
"wildcard": 2,
"match_phrase": 46896,
"terms": 1586376,
"match_all": 2,
"match_phrase_prefix": 6,
"exists": 1663023,
"term": 1852825,
"simple_query_string": 58060
},
"rescorers": {},
"sections": {
"highlight": 17,
"search_after": 15237,
"stored_fields": 34,
"runtime_mappings": 78286,
"query": 1858973,
"script_fields": 34,
"terminate_after": 11,
"pit": 37107,
"_source": 9758,
"fields": 4184,
"collapse": 36594,
"aggs": 230989
},
"retrievers": {}
},
"dense_vector": {
"value_count": 0
},
"sparse_vector": {
"value_count": 0
}
},
"nodes": {
"count": {
"total": 3,
"coordinating_only": 0,
"data": 2,
"data_cold": 2,
"data_content": 2,
"data_frozen": 2,
"data_hot": 2,
"data_warm": 2,
"index": 0,
"ingest": 2,
"master": 3,
"ml": 2,
"remote_cluster_client": 2,
"search": 0,
"transform": 2,
"voting_only": 1
},
"versions": [
"8.17.0"
],
"os": {
"available_processors": 18,
"allocated_processors": 18,
"names": [
{
"name": "Linux",
"count": 3
}
],
"pretty_names": [
{
"pretty_name": "Ubuntu 24.04.1 LTS",
"count": 3
}
],
"architectures": [
{
"arch": "amd64",
"count": 3
}
],
"mem": {
"total": "48.8gb",
"total_in_bytes": 52461871104,
"adjusted_total": "48.8gb",
"adjusted_total_in_bytes": 52461871104,
"free": "1gb",
"free_in_bytes": 1087062016,
"used": "47.8gb",
"used_in_bytes": 51374809088,
"free_percent": 2,
"used_percent": 98
}
},
"process": {
"cpu": {
"percent": 53
},
"open_file_descriptors": {
"min": 629,
"max": 3751,
"avg": 2708
}
},
"jvm": {
"max_uptime": "19d",
"max_uptime_in_millis": 1645999034,
"versions": [
{
"version": "23",
"vm_name": "OpenJDK 64-Bit Server VM",
"vm_version": "23+37-2369",
"vm_vendor": "Oracle Corporation",
"bundled_jdk": true,
"using_bundled_jdk": true,
"count": 3
}
],
"mem": {
"heap_used": "20.5gb",
"heap_used_in_bytes": 22104554928,
"heap_max": "24.9gb",
"heap_max_in_bytes": 26801602560
},
"threads": 457
},
"fs": {
"total": "1.5tb",
"total_in_bytes": 1707082899456,
"free": "755.5gb",
"free_in_bytes": 811279769600,
"available": "720.7gb",
"available_in_bytes": 773864079360
},
"plugins": [],
"network_types": {
"transport_types": {
"security4": 3
},
"http_types": {
"security4": 3
}
},
"discovery_types": {
"multi-node": 3
},
"packaging_types": [
{
"flavor": "default",
"type": "deb",
"count": 3
}
],
"ingest": {
"number_of_pipelines": 305,
"processor_stats": {
"append": {
"count": 1663342,
"failed": 0,
"current": 0,
"time": "6.5s",
"time_in_millis": 6563
},
"attachment": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"community_id": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"convert": {
"count": 1263266,
"failed": 0,
"current": 0,
"time": "2.3s",
"time_in_millis": 2305
},
"csv": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"date": {
"count": 7646722,
"failed": 0,
"current": 0,
"time": "1.6m",
"time_in_millis": 98818
},
"date_index_name": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"dissect": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"dot_expander": {
"count": 66944,
"failed": 0,
"current": 0,
"time": "1.7s",
"time_in_millis": 1756
},
"drop": {
"count": 804884,
"failed": 0,
"current": 0,
"time": "120ms",
"time_in_millis": 120
},
"enrich": {
"count": 217075328,
"failed": 3376,
"current": 0,
"time": "1.3d",
"time_in_millis": 114007477
},
"fail": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"fingerprint": {
"count": 12584063,
"failed": 0,
"current": 0,
"time": "42.9s",
"time_in_millis": 42994
},
"foreach": {
"count": 14279,
"failed": 0,
"current": 0,
"time": "206ms",
"time_in_millis": 206
},
"geoip": {
"count": 240094323,
"failed": 0,
"current": 0,
"time": "28.7m",
"time_in_millis": 1726387
},
"grok": {
"count": 10730001,
"failed": 642745,
"current": 0,
"time": "7.5m",
"time_in_millis": 450424
},
"gsub": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"inference": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"join": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"json": {
"count": 66944,
"failed": 0,
"current": 0,
"time": "1.6s",
"time_in_millis": 1613
},
"kv": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"lowercase": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"pipeline": {
"count": 403040954,
"failed": 0,
"current": 0,
"time": "3.6m",
"time_in_millis": 217691
},
"remove": {
"count": 312415699,
"failed": 0,
"current": 0,
"time": "3.8m",
"time_in_millis": 228604
},
"rename": {
"count": 253835892,
"failed": 1049908,
"current": 0,
"time": "13m",
"time_in_millis": 782411
},
"reroute": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"script": {
"count": 196616849,
"failed": 0,
"current": 0,
"time": "5.5m",
"time_in_millis": 330215
},
"set": {
"count": 22788381,
"failed": 0,
"current": 0,
"time": "1m",
"time_in_millis": 60294
},
"set_security_user": {
"count": 96265925,
"failed": 0,
"current": 0,
"time": "7.9m",
"time_in_millis": 476935
},
"split": {
"count": 2514457,
"failed": 939378,
"current": 0,
"time": "11.6s",
"time_in_millis": 11667
},
"trim": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"uppercase": {
"count": 0,
"failed": 0,
"current": 0,
"time": "0s",
"time_in_millis": 0
},
"uri_parts": {
"count": 482761,
"failed": 1400,
"current": 0,
"time": "2.8s",
"time_in_millis": 2881
},
"user_agent": {
"count": 3214860,
"failed": 0,
"current": 0,
"time": "13.6s",
"time_in_millis": 13673
}
}
},
"indexing_pressure": {
"memory": {
"current": {
"combined_coordinating_and_primary": "0b",
"combined_coordinating_and_primary_in_bytes": 0,
"coordinating": "0b",
"coordinating_in_bytes": 0,
"primary": "0b",
"primary_in_bytes": 0,
"replica": "0b",
"replica_in_bytes": 0,
"all": "0b",
"all_in_bytes": 0
},
"total": {
"combined_coordinating_and_primary": "0b",
"combined_coordinating_and_primary_in_bytes": 0,
"coordinating": "0b",
"coordinating_in_bytes": 0,
"primary": "0b",
"primary_in_bytes": 0,
"replica": "0b",
"replica_in_bytes": 0,
"all": "0b",
"all_in_bytes": 0,
"coordinating_rejections": 0,
"primary_rejections": 0,
"replica_rejections": 0,
"primary_document_rejections": 0
},
"limit": "0b",
"limit_in_bytes": 0
}
}
},
"snapshots": {
"current_counts": {
"snapshots": 0,
"shard_snapshots": 0,
"snapshot_deletions": 0,
"concurrent_operations": 0,
"cleanups": 0
},
"repositories": {
"s3-ovh": {
"type": "s3",
"current_counts": {
"snapshots": 0,
"clones": 0,
"finalizations": 0,
"deletions": 0,
"snapshot_deletions": 0,
"active_deletions": 0,
"shards": {
"total": 0,
"complete": 0,
"incomplete": 0,
"states": {
"INIT": 0,
"SUCCESS": 0,
"FAILED": 0,
"ABORTED": 0,
"MISSING": 0,
"WAITING": 0,
"QUEUED": 0,
"PAUSED_FOR_NODE_REMOVAL": 0
}
}
},
"oldest_start_time": "2025-01-08T10:38:30.276Z",
"oldest_start_time_millis": 1736332710276
}
}
},
"repositories": {
"s3": {
"count": 1,
"read_write": 1
}
},
"ccs": {
"_search": {
"total": 0,
"success": 0,
"skipped": 0,
"took": {
"max": 0,
"avg": 0,
"p90": 0
},
"took_mrt_true": {
"max": 0,
"avg": 0,
"p90": 0
},
"took_mrt_false": {
"max": 0,
"avg": 0,
"p90": 0
},
"remotes_per_search_max": 0,
"remotes_per_search_avg": 0,
"failure_reasons": {},
"features": {},
"clients": {},
"clusters": {}
}
}
}
Any idea what I could check to see what could be the cause of this issue ?