Indexing rate became very low in elastic cluster

Subrahmanyam_Veerank · February 2, 2024, 9:08am

The indexing rate of our cluster become very low and even search results are getting delayed.
cluster details:

3 Master nodes - cpu 4 cores -RAM 16GB
4 Data nodes (out of 4 one node is given both ingesting and data node role) - CPU 8 cores- RAM 30GB

As of now we have configured 16 agents (linux machines and kubernetes nodes)
with integrations nginx, kubernetes and ingress controller.
But the logs are getting ingested at low rate and it was lagging with the current time.
what are the main causes for this issue? Kindly suggest any modifications that are required.

Christian_Dahlqvist · February 2, 2024, 9:14am

I would recommend configuring all data nodes to also be ingest nodes if you are using ingest pipeline. With your current configuration you can easily get a very unbalanced load.

What is the full output of the cluster stats API?

Are you directing all indexing and query load to the data nodes?

How many indices and shards are you actively indexing into?

What type of storage are you using? Have you checked what await and disk utilisation looks like on the various nodes? Slow storage is a common source of performance problems.

Do you see any evidence in the logs of long or frequent GC on any of the nodes?

Subrahmanyam_Veerank · February 2, 2024, 12:05pm

we have configured the ingest role to all the 4 data nodes.

The full output of the cluster stats is below:
{
"_nodes": {
"total": 7,
"successful": 7,
"failed": 0
},
"cluster_name": "es-prod1",
"cluster_uuid": "XJzleHF2QWsVL5fgY-A",
"timestamp": 1706874280018,
"status": "green",
"indices": {
"count": 132,
"shards": {
"total": 265,
"primaries": 132,
"replication": 1.0075757575757576,
"index": {
"shards": {
"min": 2,
"max": 3,
"avg": 2.007575757575758
},
"primaries": {
"min": 1,
"max": 1,
"avg": 1
},
"replication": {
"min": 1,
"max": 2,
"avg": 1.0075757575757576
}
}
},
"docs": {
"count": 984868250,
"deleted": 62399
},
"store": {
"size_in_bytes": 459265536546,
"total_data_set_size_in_bytes": 459265536546,
"reserved_in_bytes": 0
},
"fielddata": {
"memory_size_in_bytes": 255824,
"evictions": 0,
"global_ordinals": {
"build_time_in_millis": 212505
}
},
"query_cache": {
"memory_size_in_bytes": 120682104,
"total_count": 4091929,
"hit_count": 752382,
"miss_count": 3339547,
"cache_size": 19934,
"cache_count": 34403,
"evictions": 14469
},
"completion": {
"size_in_bytes": 0
},
"segments": {
"count": 5598,
"memory_in_bytes": 0,
"terms_memory_in_bytes": 0,
"stored_fields_memory_in_bytes": 0,
"term_vectors_memory_in_bytes": 0,
"norms_memory_in_bytes": 0,
"points_memory_in_bytes": 0,
"doc_values_memory_in_bytes": 0,
"index_writer_memory_in_bytes": 31201600,
"version_map_memory_in_bytes": 449993,
"fixed_bit_set_memory_in_bytes": 243048,
"max_unsafe_auto_id_timestamp": 1706872889053,
"file_sizes": {}
},
"mappings": {
"total_field_count": 20553,
"total_deduplicated_field_count": 15533,
"total_deduplicated_mapping_size_in_bytes": 130167,
"field_types": [
{
"name": "aggregate_metric_double",
"count": 18,
"index_count": 9,
"script_count": 0
},
{
"name": "alias",
"count": 77,
"index_count": 2,
"script_count": 0
},
{
"name": "binary",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "boolean",
"count": 333,
"index_count": 94,
"script_count": 0
},
{
"name": "byte",
"count": 4,
"index_count": 4,
"script_count": 0
},
{
"name": "constant_keyword",
"count": 293,
"index_count": 72,
"script_count": 0
},
{
"name": "date",
"count": 609,
"index_count": 110,
"script_count": 0
},
{
"name": "date_range",
"count": 9,
"index_count": 9,
"script_count": 0
},
{
"name": "double",
"count": 32,
"index_count": 10,
"script_count": 0
},
{
"name": "flattened",
"count": 54,
"index_count": 10,
"script_count": 0
},
{
"name": "float",
"count": 162,
"index_count": 24,
"script_count": 0
},
{
"name": "geo_point",
"count": 41,
"index_count": 13,
"script_count": 0
},
{
"name": "half_float",
"count": 56,
"index_count": 14,
"script_count": 0
},
{
"name": "histogram",
"count": 10,
"index_count": 10,
"script_count": 0
},
{
"name": "integer",
"count": 159,
"index_count": 10,
"script_count": 0
},
{
"name": "ip",
"count": 165,
"index_count": 64,
"script_count": 0
},
{
"name": "keyword",
"count": 9550,
"index_count": 111,
"script_count": 0
},
{
"name": "long",
"count": 3283,
"index_count": 101,
"script_count": 0
},
{
"name": "match_only_text",
"count": 490,
"index_count": 60,
"script_count": 0
},
{
"name": "nested",
"count": 76,
"index_count": 14,
"script_count": 0
},
{
"name": "object",
"count": 4842,
"index_count": 108,
"script_count": 0
},
{
"name": "rank_features",
"count": 1,
"index_count": 1,
"script_count": 0
},
{
"name": "scaled_float",
"count": 102,
"index_count": 17,
"script_count": 0
},
{
"name": "text",
"count": 72,
"index_count": 27,
"script_count": 0
},
{
"name": "version",
"count": 12,
"index_count": 12,
"script_count": 0
},
{
"name": "wildcard",
"count": 102,
"index_count": 12,
"script_count": 0
}
],
"runtime_field_types":
},
"analysis": {
"char_filter_types": ,
"tokenizer_types": ,
"filter_types": ,
"analyzer_types": ,
"built_in_char_filters": ,
"built_in_tokenizers": ,
"built_in_filters": ,
"built_in_analyzers": ,
"synonyms": {}
},
"versions": [
{
"version": "8500003",
"index_count": 132,
"primary_shard_count": 132,
"total_primary_bytes": 233318490076
}
],
"search": {
"total": 1360754,
"queries": {
"geo_bounding_box": 302,
"bool": 1289378,
"prefix": 31,
"match": 220369,
"range": 770559,
"nested": 13784,
"multi_match": 169,
"wildcard": 51,
"match_phrase": 16313,
"terms": 543461,
"constant_score": 1,
"match_all": 249,
"match_phrase_prefix": 871,
"exists": 440565,
"term": 1060005,
"query_string": 4847,
"simple_query_string": 32207
},
"sections": {
"search_after": 71,
"highlight": 714,
"stored_fields": 14501,
"runtime_mappings": 199253,
"query": 1323976,
"script_fields": 14501,
"_source": 62855,
"pit": 8509,
"terminate_after": 1430,
"fields": 31149,
"collapse": 164501,
"aggs": 441197
}
},
"dense_vector": {
"value_count": 0
}
},
"nodes": {
"count": {
"total": 7,
"coordinating_only": 0,
"data": 4,
"data_cold": 0,
"data_content": 0,
"data_frozen": 0,
"data_hot": 0,
"data_warm": 0,
"index": 0,
"ingest": 4,
"master": 3,
"ml": 1,
"remote_cluster_client": 4,
"search": 0,
"transform": 1,
"voting_only": 0
},
"versions": [
"8.11.4"
],
"os": {
"available_processors": 44,
"allocated_processors": 44,
"names": [
{
"name": "Linux",
"count": 7
}
],
"pretty_names": [
{
"pretty_name": "Ubuntu 22.04.3 LTS",
"count": 7
}
],
"architectures": [
{
"arch": "amd64",
"count": 7
}
],
"mem": {
"total_in_bytes": 150866984960,
"adjusted_total_in_bytes": 150866984960,
"free_in_bytes": 14318759936,
"used_in_bytes": 136548225024,
"free_percent": 9,
"used_percent": 91
}
},
"process": {
"cpu": {
"percent": 13
},
"open_file_descriptors": {
"min": 749,
"max": 1060,
"avg": 880
}
},
"jvm": {
"max_uptime_in_millis": 1532768998,
"versions": [
{
"version": "21.0.1",
"vm_name": "OpenJDK 64-Bit Server VM",
"vm_version": "21.0.1+12-29",
"vm_vendor": "Oracle Corporation",
"bundled_jdk": true,
"using_bundled_jdk": true,
"count": 7
}
],
"mem": {
"heap_used_in_bytes": 21972146384,
"heap_max_in_bytes": 75447140352
},
"threads": 697
},
"fs": {
"total_in_bytes": 1990908674048,
"free_in_bytes": 1456145158144,
"available_in_bytes": 1371346018304
},
"plugins": ,
"network_types": {
"transport_types": {
"security4": 7
},
"http_types": {
"security4": 7
}
},
"discovery_types": {
"multi-node": 7
},
"packaging_types": [
{
"flavor": "default",
"type": "deb",
"count": 7
}
],
"ingest": {
"number_of_pipelines": 151,
"processor_stats": {
"append": {
"count": 377811919,
"failed": 0,
"current": 0,
"time_in_millis": 710044
},
"attachment": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"community_id": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"convert": {
"count": 78790959,
"failed": 2,
"current": 0,
"time_in_millis": 146093
},
"date": {
"count": 148545191,
"failed": 0,
"current": 0,
"time_in_millis": 1861910
},
"date_index_name": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"dot_expander": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"drop": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"fail": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"fingerprint": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"foreach": {
"count": 1758,
"failed": 0,
"current": 0,
"time_in_millis": 30
},
"geoip": {
"count": 225910147,
"failed": 0,
"current": 0,
"time_in_millis": 85082
},
"grok": {
"count": 461526527,
"failed": 231355,
"current": 0,
"time_in_millis": 19188553
},
"gsub": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"inference": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"join": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"json": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"kv": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"lowercase": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"pipeline": {
"count": 980611583,
"failed": 3,
"current": 0,
"time_in_millis": 4970023
},
"remove": {
"count": 3276924626,
"failed": 0,
"current": 0,
"time_in_millis": 2256235
},
"rename": {
"count": 520036113,
"failed": 14020965,
"current": 0,
"time_in_millis": 744968
},
"reroute": {
"count": 612938045,
"failed": 0,
"current": 0,
"time_in_millis": 5077490
},
"script": {
"count": 2262651108,
"failed": 0,
"current": 0,
"time_in_millis": 5795674
},
"set": {
"count": 2478270959,
"failed": 0,
"current": 0,
"time_in_millis": 4065633
},
"set_security_user": {
"count": 980796848,
"failed": 0,
"current": 0,
"time_in_millis": 5251716
},
"split": {
"count": 300866118,
"failed": 0,
"current": 0,
"time_in_millis": 805278
},
"trim": {
"count": 0,
"failed": 0,
"current": 0,
"time_in_millis": 0
},
"uri_parts": {
"count": 112910485,
"failed": 1729,
"current": 0,
"time_in_millis": 259562
},
"user_agent": {
"count": 112910592,
"failed": 0,
"current": 0,
"time_in_millis": 265202
}
}
},
"indexing_pressure": {
"memory": {
"current": {
"combined_coordinating_and_primary_in_bytes": 0,
"coordinating_in_bytes": 0,
"primary_in_bytes": 0,
"replica_in_bytes": 0,
"all_in_bytes": 0
},
"total": {
"combined_coordinating_and_primary_in_bytes": 0,
"coordinating_in_bytes": 0,
"primary_in_bytes": 0,
"replica_in_bytes": 0,
"all_in_bytes": 0,
"coordinating_rejections": 0,
"primary_rejections": 0,
"replica_rejections": 0
},
"limit_in_bytes": 0
}
}
},
"snapshots": {
"current_counts": {
"snapshots": 0,
"shard_snapshots": 0,
"snapshot_deletions": 0,
"concurrent_operations": 0,
"cleanups": 0
},
"repositories": {}
}
}
Currently all the indices are created on the data nodes only and no index is on master node.
133 indices and 265 shards (out of 265 shards - primary 132 ) are currently indexing.
we are using the SAN HDD type of storage.
what we need to check in the gc.log. Below root@ELK04:/home/posttech# tail -300f /var/log/ela [2024-02-02T11:11:13.621+0000][7708][info ][gc,phases [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][trace][gc,age [2024-02-02T11:11:13.621+0000][7708][info ][gc,heap [2024-02-02T11:11:13.621+0000][7708][info ][gc,heap [2024-02-02T11:11:13.621+0000][7708][info ][gc,heap [2024-02-02T11:11:13.621+0000][7708][info ][gc,heap [2024-02-02T11:11:13.621+0000][7708][info [2024-02-02T11:11:13.621+0000][7708][info ][gc [2024-02-02T11:11:13.621+0000][7708][info ][gc,cpu [2024-02-02T11:11:13.621+0000][7708][info ][safepoint [2024-02-02T11:12:26.090+0000][7708][info ][safepoint [2024-02-02T11:14:05.834+0000][7708][info ][safepoint [2024-02-02T11:14:13.382+0000][7708][info ][gc,start [2024-02-02T11:14:13.382+0000][7708][info ][gc,task [2024-02-02T11:14:13.382+0000][7708][debug][gc,age [2024-02-02T11:14:13.445+0000][7708][info ][gc,phases [2024-02-02T11:14:13.445+0000][7708][info ][gc,phases [2024-02-02T11:14:13.445+0000][7708][info ][gc,phases [2024-02-02T11:14:13.445+0000][7708][info ][gc,phases [2024-02-02T11:14:13.445+0000][7708][info ][gc,phases [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][trace][gc,age [2024-02-02T11:14:13.445+0000][7708][info ][gc,heap [2024-02-02T11:14:13.445+0000][7708][info ][gc,heap [2024-02-02T11:14:13.445+0000][7708][info ][gc,heap [2024-02-02T11:14:13.445+0000][7708][info ][gc,heap [2024-02-02T11:14:13.445+0000][7708][info [2024-02-02T11:14:13.445+0000][7708][info ][gc [2024-02-02T11:14:13.446+0000][7708][info ][gc,cpu [2024-02-02T11:14:13.446+0000][7708][info ][safepoint [2024-02-02T11:15:27.080+0000][7708][info ][safepoint [2024-02-02T11:16:17.718+0000][7708][info ][safepoint [2024-02-02T11:17:10.476+0000][7708][info ][gc,start [2024-02-02T11:17:10.476+0000][7708][info ][gc,task [2024-02-02T11:17:10.476+0000][7708][debug][gc,age [2024-02-02T11:17:10.537+0000][7708][info ][gc,phases [2024-02-02T11:17:10.537+0000][7708][info ][gc,phases [2024-02-02T11:17:10.537+0000][7708][info ][gc,phases [2024-02-02T11:17:10.537+0000][7708][info ][gc,phases [2024-02-02T11:17:10.537+0000][7708][info ][gc,phases [2024-02-02T11:17:10.537+0000][7708][trace][gc,age [2024-02-02T11:17:10.537+0000][7708][trace][gc,age [2024-02-02T11:17:10.537+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][trace][gc,age [2024-02-02T11:17:10.538+0000][7708][info ][gc,heap [2024-02-02T11:17:10.538+0000][7708][info ][gc,heap [2024-02-02T11:17:10.538+0000][7708][info ][gc,heap [2024-02-02T11:17:10.538+0000][7708][info ][gc,heap [2024-02-02T11:17:10.538+0000][7708][info [2024-02-02T11:17:10.538+0000][7708][info ][gc [2024-02-02T11:17:10.538+0000][7708][info ][gc,cpu [2024-02-02T11:17:10.538+0000][7708][info ][safepoint [2024-02-02T11:17:28.597+0000][7708][info ][safepoint [2024-02-02T11:17:42.483+0000][7708][info ][safepoint [2024-02-02T11:18:34.770+0000][7708][info ][safepoint [2024-02-02T11:18:36.159+0000][7708][info ][safepoint [2024-02-02T11:18:47.660+0000][7708][info ][safepoint [2024-02-02T11:19:28.563+0000][7708][info ][safepoint [2024-02-02T11:22:59.993+0000][7708][info ][gc,start [2024-02-02T11:22:59.993+0000][7708][info ][gc,task [2024-02-02T11:22:59.993+0000][7708][debug][gc,age [2024-02-02T11:23:00.066+0000][7708][info ][gc,phases [2024-02-02T11:23:00.066+0000][7708][info ][gc,phases [2024-02-02T11:23:00.066+0000][7708][info ][gc,phases [2024-02-02T11:23:00.066+0000][7708][info ][gc,phases [2024-02-02T11:23:00.066+0000][7708][info ][gc,phases [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][trace][gc,age [2024-02-02T11:23:00.066+0000][7708][info ][gc,heap [2024-02-02T11:23:00.066+0000][7708][info ][gc,heap [2024-02-02T11:23:00.066+0000][7708][info ][gc,heap [2024-02-02T11:23:00.066+0000][7708][info ][gc,heap [2024-02-02T11:23:00.066+0000][7708][info [2024-02-02T11:23:00.066+0000][7708][info ][gc [2024-02-02T11:23:00.066+0000][7708][info ][gc,cpu [2024-02-02T11:23:00.066+0000][7708][info ][safepoint [2024-02-02T11:25:27.121+0000][7708][info ][safepoint [2024-02-02T11:26:40.096+0000][7708][info ][safepoint [2024-02-02T11:27:54.988+0000][7708][info ][gc,start [2024-02-02T11:27:54.988+0000][7708][info ][gc,task [2024-02-02T11:27:54.988+0000][7708][debug][gc,age [2024-02-02T11:27:55.061+0000][7708][info ][gc,phases [2024-02-02T11:27:55.061+0000][7708][info ][gc,phases [2024-02-02T11:27:55.061+0000][7708][info ][gc,phases [2024-02-02T11:27:55.061+0000][7708][info ][gc,phases [2024-02-02T11:27:55.061+0000][7708][info ][gc,phases [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][trace][gc,age [2024-02-02T11:27:55.061+0000][7708][info ][gc,heap [2024-02-02T11:27:55.061+0000][7708][info ][gc,heap [2024-02-02T11:27:55.061+0000][7708][info ][gc,heap [2024-02-02T11:27:55.061+0000][7708][info ][gc,heap [2024-02-02T11:27:55.061+0000][7708][info [2024-02-02T11:27:55.062+0000][7708][info ][gc [2024-02-02T11:27:55.062+0000][7708][info ][gc,cpu [2024-02-02T11:27:55.062+0000][7708][info ][safepoint [2024-02-02T11:32:33.110+0000][7708][info ][safepoint [2024-02-02T11:33:34.302+0000][7708][info ][safepoint [2024-02-02T11:33:59.534+0000][7708][info ][gc,start [2024-02-02T11:33:59.534+0000][7708][info ][gc,task [2024-02-02T11:33:59.534+0000][7708][debug][gc,age [2024-02-02T11:33:59.610+0000][7708][info ][gc,phases [2024-02-02T11:33:59.610+0000][7708][info ][gc,phases [2024-02-02T11:33:59.610+0000][7708][info ][gc,phases [2024-02-02T11:33:59.610+0000][7708][info ][gc,phases [2024-02-02T11:33:59.610+0000][7708][info ][gc,phases [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][trace][gc,age [2024-02-02T11:33:59.610+0000][7708][info ][gc,heap [2024-02-02T11:33:59.610+0000][7708][info ][gc,heap [2024-02-02T11:33:59.610+0000][7708][info ][gc,heap [2024-02-02T11:33:59.610+0000][7708][info ][gc,heap [2024-02-02T11:33:59.610+0000][7708][info [2024-02-02T11:33:59.610+0000][7708][info ][gc [2024-02-02T11:33:59.610+0000][7708][info ][gc,cpu [2024-02-02T11:33:59.610+0000][7708][info ][safepoint [2024-02-02T11:39:13.201+0000][7708][info ][safepoint [2024-02-02T11:41:31.031+0000][7708][info ][gc,start [2024-02-02T11:41:31.032+0000][7708][info ][gc,task [2024-02-02T11:41:31.032+0000][7708][debug][gc,age [2024-02-02T11:41:31.112+0000][7708][info ][gc,phases [2024-02-02T11:41:31.112+0000][7708][info ][gc,phases [2024-02-02T11:41:31.112+0000][7708][info ][gc,phases [2024-02-02T11:41:31.112+0000][7708][info ][gc,phases [2024-02-02T11:41:31.112+0000][7708][info ][gc,phases [2024-02-02T11:41:31.112+0000][7708][trace][gc,age [2024-02-02T11:41:31.112+0000][7708][trace][gc,age [2024-02-02T11:41:31.112+0000][7708][trace][gc,age [2024-02-02T11:41:31.112+0000][7708][trace][gc,age [2024-02-02T11:41:31.112+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][trace][gc,age [2024-02-02T11:41:31.113+0000][7708][info ][gc,heap [2024-02-02T11:41:31.113+0000][7708][info ][gc,heap [2024-02-02T11:41:31.113+0000][7708][info ][gc,heap [2024-02-02T11:41:31.113+0000][7708][info ][gc,heap [2024-02-02T11:41:31.113+0000][7708][info [2024-02-02T11:41:31.113+0000][7708][info ][gc [2024-02-02T11:41:31.113+0000][7708][info ][gc,cpu [2024-02-02T11:41:31.113+0000][7708][info ][safepoint [2024-02-02T11:45:47.882+0000][7708][info ][safepoint [2024-02-02T11:46:57.818+0000][7708][info ][safepoint [2024-02-02T11:49:03.274+0000][7708][info ][gc,start [2024-02-02T11:49:03.274+0000][7708][info ][gc,task [2024-02-02T11:49:03.274+0000][7708][debug][gc,age [2024-02-02T11:49:03.343+0000][7708][info ][gc,phases [2024-02-02T11:49:03.343+0000][7708][info ][gc,phases [2024-02-02T11:49:03.343+0000][7708][info ][gc,phases [2024-02-02T11:49:03.343+0000][7708][info ][gc,phases [2024-02-02T11:49:03.343+0000][7708][info ][gc,phases [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][trace][gc,age [2024-02-02T11:49:03.343+0000][7708][info ][gc,heap [2024-02-02T11:49:03.343+0000][7708][info ][gc,heap [2024-02-02T11:49:03.343+0000][7708][info ][gc,heap [2024-02-02T11:49:03.343+0000][7708][info ][gc,heap [2024-02-02T11:49:03.343+0000][7708][info [2024-02-02T11:49:03.343+0000][7708][info ][gc [2024-02-02T11:49:03.343+0000][7708][info ][gc,cpu [2024-02-02T11:49:03.343+0000][7708][info ][safepoint [2024-02-02T11:51:52.405+0000][7708][info ][safepoint [2024-02-02T11:55:17.656+0000][7708][info ][safepoint [2024-02-02T11:55:54.808+0000][7708][info ][gc,start [2024-02-02T11:55:54.808+0000][7708][info ][gc,task [2024-02-02T11:55:54.808+0000][7708][debug][gc,age [2024-02-02T11:55:54.903+0000][7708][info ][gc,phases [2024-02-02T11:55:54.903+0000][7708][info ][gc,phases [2024-02-02T11:55:54.903+0000][7708][info ][gc,phases [2024-02-02T11:55:54.903+0000][7708][info ][gc,phases [2024-02-02T11:55:54.903+0000][7708][info ][gc,phases [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][trace][gc,age [2024-02-02T11:55:54.903+0000][7708][info ][gc,heap [2024-02-02T11:55:54.903+0000][7708][info ][gc,heap [2024-02-02T11:55:54.903+0000][7708][info ][gc,heap [2024-02-02T11:55:54.903+0000][7708][info ][gc,heap [2024-02-02T11:55:54.903+0000][7708][info [2024-02-02T11:55:54.904+0000][7708][info ][gc [2024-02-02T11:55:54.904+0000][7708][info ][gc,cpu [2024-02-02T11:55:54.904+0000][7708][info ][safepoint [2024-02-02T11:56:54.631+0000][7708][info ][safepoint [2024-02-02T11:56:55.834+0000][7708][info ][safepoint is the sample gc log in node4
sticsearch/gc.log
] GC(11434) Other: 0.1ms
] GC(11434) Age table with threshold 15 (max threshold 15)
] GC(11434) - age 1: 163152688 bytes, 163152688 total
] GC(11434) - age 2: 17799408 bytes, 180952096 total
] GC(11434) - age 3: 10933792 bytes, 191885888 total
] GC(11434) - age 4: 2062032 bytes, 193947920 total
] GC(11434) - age 5: 1946368 bytes, 195894288 total
] GC(11434) - age 6: 6880880 bytes, 202775168 total
] GC(11434) - age 7: 7177872 bytes, 209953040 total
] GC(11434) - age 8: 6060304 bytes, 216013344 total
] GC(11434) - age 9: 3087064 bytes, 219100408 total
] GC(11434) - age 10: 6869008 bytes, 225969416 total
] GC(11434) - age 11: 5823872 bytes, 231793288 total
] GC(11434) - age 12: 9165880 bytes, 240959168 total
] GC(11434) - age 13: 2018128 bytes, 242977296 total
] GC(11434) - age 14: 451256 bytes, 243428552 total
] GC(11434) - age 15: 2750448 bytes, 246179000 total
] GC(11434) Eden regions: 1107->0(1095)
] GC(11434) Survivor regions: 19->31(141)
] GC(11434) Old regions: 174->174
] GC(11434) Humongous regions: 7->7
][gc,metaspace] GC(11434) Metaspace: 199655K(212928K)->199655K(212928K) NonClass: 173301K(184128K)->173301K(184128K) Class: 26354K(28800K)->26354K(28800K)
] GC(11434) Pause Young (Normal) (G1 Evacuation Pause) 10446M->1690M(15024M) 54.136ms
] GC(11434) User=0.38s Sys=0.01s Real=0.05s
] Safepoint "G1CollectForAllocation", Time since last: 14091793600 ns, Reaching safepoint: 15457300 ns, Cleanup: 13700 ns, At safepoint: 54267700 ns, Total: 69738700 ns
] Safepoint "Cleanup", Time since last: 72468409100 ns, Reaching safepoint: 15800 ns, Cleanup: 15300 ns, At safepoint: 8700 ns, Total: 39800 ns
] Safepoint "Cleanup", Time since last: 99743981900 ns, Reaching safepoint: 20300 ns, Cleanup: 13000 ns, At safepoint: 9700 ns, Total: 43000 ns
] GC(11435) Pause Young (Normal) (G1 Evacuation Pause)
] GC(11435) Using 8 workers of 8 for evacuation
] GC(11435) Desired survivor size 591396864 bytes, new threshold 15 (max threshold 15)
] GC(11435) Pre Evacuate Collection Set: 0.7ms
] GC(11435) Merge Heap Roots: 0.6ms
] GC(11435) Evacuate Collection Set: 57.1ms
] GC(11435) Post Evacuate Collection Set: 4.4ms
] GC(11435) Other: 0.3ms
] GC(11435) Age table with threshold 15 (max threshold 15)
] GC(11435) - age 1: 61354752 bytes, 61354752 total
] GC(11435) - age 2: 10819848 bytes, 72174600 total
] GC(11435) - age 3: 15803584 bytes, 87978184 total
] GC(11435) - age 4: 10236056 bytes, 98214240 total
] GC(11435) - age 5: 1384848 bytes, 99599088 total
] GC(11435) - age 6: 1377552 bytes, 100976640 total
] GC(11435) - age 7: 6474536 bytes, 107451176 total
] GC(11435) - age 8: 6858264 bytes, 114309440 total
] GC(11435) - age 9: 5732496 bytes, 120041936 total
] GC(11435) - age 10: 1032848 bytes, 121074784 total
] GC(11435) - age 11: 6541456 bytes, 127616240 total
] GC(11435) - age 12: 5806768 bytes, 133423008 total
] GC(11435) - age 13: 9165496 bytes, 142588504 total
] GC(11435) - age 14: 2018064 bytes, 144606568 total
] GC(11435) - age 15: 451128 bytes, 145057696 total
] GC(11435) Eden regions: 1095->0(1106)
] GC(11435) Survivor regions: 31->20(141)
] GC(11435) Old regions: 174->174
] GC(11435) Humongous regions: 7->7
][gc,metaspace] GC(11435) Metaspace: 199661K(212928K)->199661K(212928K) NonClass: 173306K(184128K)->173306K(184128K) Class: 26354K(28800K)->26354K(28800K)
] GC(11435) Pause Young (Normal) (G1 Evacuation Pause) 10450M->1601M(15024M) 63.407ms
] GC(11435) User=0.45s Sys=0.00s Real=0.07s
] Safepoint "G1CollectForAllocation", Time since last: 7548056500 ns, Reaching safepoint: 31600 ns, Cleanup: 24600 ns, At safepoint: 63560900 ns, Total: 63617100 ns
] Safepoint "Cleanup", Time since last: 73634049900 ns, Reaching safepoint: 142700 ns, Cleanup: 21200 ns, At safepoint: 79900 ns, Total: 243800 ns
] Safepoint "Cleanup", Time since last: 50638326100 ns, Reaching safepoint: 15400 ns, Cleanup: 22100 ns, At safepoint: 9100 ns, Total: 46600 ns
] GC(11436) Pause Young (Normal) (G1 Evacuation Pause)
] GC(11436) Using 8 workers of 8 for evacuation
] GC(11436) Desired survivor size 591396864 bytes, new threshold 15 (max threshold 15)
] GC(11436) Pre Evacuate Collection Set: 0.7ms
] GC(11436) Merge Heap Roots: 0.7ms
] GC(11436) Evacuate Collection Set: 55.0ms
] GC(11436) Post Evacuate Collection Set: 4.3ms
] GC(11436) Other: 0.3ms
] GC(11436) Age table with threshold 15 (max threshold 15)
] GC(11436) - age 1: 55568432 bytes, 55568432 total
] GC(11436) - age 2: 15571808 bytes, 71140240 total
] GC(11436) - age 3: 9735776 bytes, 80876016 total
] GC(11436) - age 4: 15036248 bytes, 95912264 total
] GC(11436) - age 5: 10235264 bytes, 106147528 total
] GC(11436) - age 6: 1384112 bytes, 107531640 total
] GC(11436) - age 7: 1377424 bytes, 108909064 total
] GC(11436) - age 8: 6473480 bytes, 115382544 total
] GC(11436) - age 9: 6857832 bytes, 122240376 total
] GC(11436) - age 10: 5625728 bytes, 127866104 total
] GC(11436) - age 11: 1023000 bytes, 128889104 total
] GC(11436) - age 12: 6541328 bytes, 135430432 total
] GC(11436) - age 13: 5806736 bytes, 141237168 total
] GC(11436) - age 14: 9165432 bytes, 150402600 total
] GC(11436) - age 15: 2017040 bytes, 152419640 total
] GC(11436) Eden regions: 1106->0(1105)
] GC(11436) Survivor regions: 20->21(141)
] GC(11436) Old regions: 174->174
] GC(11436) Humongous regions: 7->7
][gc,metaspace] GC(11436) Metaspace: 199661K(212928K)->199661K(212928K) NonClass: 173306K(184128K)->173306K(184128K) Class: 26354K(28800K)->26354K(28800K)
] GC(11436) Pause Young (Normal) (G1 Evacuation Pause) 10449M->1610M(15024M) 61.292ms
] GC(11436) User=0.45s Sys=0.00s Real=0.06s
] Safepoint "G1CollectForAllocation", Time since last: 52757995500 ns, Reaching safepoint: 31700 ns, Cleanup: 7200 ns, At safepoint: 61429300 ns, Total: 61468200 ns
] Safepoint "Cleanup", Time since last: 18059669300 ns, Reaching safepoint: 16000 ns, Cleanup: 12300 ns, At safepoint: 8500 ns, Total: 36800 ns
] Safepoint "Cleanup", Time since last: 13885413700 ns, Reaching safepoint: 8900 ns, Cleanup: 9000 ns, At safepoint: 3700 ns, Total: 21600 ns
] Safepoint "Cleanup", Time since last: 52287480900 ns, Reaching safepoint: 141600 ns, Cleanup: 8100 ns, At safepoint: 19900 ns, Total: 169600 ns
] Safepoint "Cleanup", Time since last: 1388677400 ns, Reaching safepoint: 13200 ns, Cleanup: 12400 ns, At safepoint: 6800 ns, Total: 32400 ns
] Safepoint "Cleanup", Time since last: 11500486300 ns, Reaching safepoint: 19000 ns, Cleanup: 11600 ns, At safepoint: 4500 ns, Total: 35100 ns
] Safepoint "Cleanup", Time since last: 40903450600 ns, Reaching safepoint: 87900 ns, Cleanup: 13700 ns, At safepoint: 12200 ns, Total: 113800 ns
] GC(11437) Pause Young (Normal) (G1 Evacuation Pause)
] GC(11437) Using 8 workers of 8 for evacuation
] GC(11437) Desired survivor size 591396864 bytes, new threshold 15 (max threshold 15)
] GC(11437) Pre Evacuate Collection Set: 0.6ms
] GC(11437) Merge Heap Roots: 0.4ms
] GC(11437) Evacuate Collection Set: 63.5ms
] GC(11437) Post Evacuate Collection Set: 8.2ms
] GC(11437) Other: 0.1ms
] GC(11437) Age table with threshold 15 (max threshold 15)
] GC(11437) - age 1: 70372568 bytes, 70372568 total
] GC(11437) - age 2: 8346208 bytes, 78718776 total
] GC(11437) - age 3: 13815824 bytes, 92534600 total
] GC(11437) - age 4: 7443248 bytes, 99977848 total
] GC(11437) - age 5: 12721704 bytes, 112699552 total
] GC(11437) - age 6: 8638616 bytes, 121338168 total
] GC(11437) - age 7: 1372648 bytes, 122710816 total
] GC(11437) - age 8: 1364976 bytes, 124075792 total
] GC(11437) - age 9: 6447616 bytes, 130523408 total
] GC(11437) - age 10: 6789272 bytes, 137312680 total
] GC(11437) - age 11: 5536296 bytes, 142848976 total
] GC(11437) - age 12: 956624 bytes, 143805600 total
] GC(11437) - age 13: 6494952 bytes, 150300552 total
] GC(11437) - age 14: 5782888 bytes, 156083440 total
] GC(11437) - age 15: 9127736 bytes, 165211176 total
] GC(11437) Eden regions: 1105->0(1104)
] GC(11437) Survivor regions: 21->22(141)
] GC(11437) Old regions: 174->175
] GC(11437) Humongous regions: 7->7
][gc,metaspace] GC(11437) Metaspace: 199667K(212928K)->199667K(212928K) NonClass: 173312K(184128K)->173312K(184128K) Class: 26354K(28800K)->26354K(28800K)
] GC(11437) Pause Young (Normal) (G1 Evacuation Pause) 10450M->1620M(15024M) 73.127ms
] GC(11437) User=0.53s Sys=0.00s Real=0.08s
] Safepoint "G1CollectForAllocation", Time since last: 211429900800 ns, Reaching safepoint: 13600 ns, Cleanup: 3600 ns, At safepoint: 73266800 ns, Total: 73284000 ns
] Safepoint "Cleanup", Time since last: 147054616500 ns, Reaching safepoint: 34100 ns, Cleanup: 19200 ns, At safepoint: 10200 ns, Total: 63500 ns
] Safepoint "Cleanup", Time since last: 72974520000 ns, Reaching safepoint: 12900 ns, Cleanup: 7900 ns, At safepoint: 8400 ns, Total: 29200 ns
] GC(11438) Pause Young (Normal) (G1 Evacuation Pause)
] GC(11438) Using 8 workers of 8 for evacuation
] GC(11438) Desired survivor size 591396864 bytes, new threshold 15 (max threshold 15)
] GC(11438) Pre Evacuate Collection Set: 0.5ms
] GC(11438) Merge Heap Roots: 0.4ms
] GC(11438) Evacuate Collection Set: 60.8ms
] GC(11438) Post Evacuate Collection Set: 11.2ms
] GC(11438) Other: 0.1ms
] GC(11438) Age table with threshold 15 (max threshold 15)
] GC(11438) - age 1: 76253224 bytes, 76253224 total
] GC(11438) - age 2: 8837344 bytes, 85090568 total
] GC(11438) - age 3: 3325304 bytes, 88415872 total
] GC(11438) - age 4: 5850728 bytes, 94266600 total
] GC(11438) - age 5: 4457048 bytes, 98723648 total
] GC(11438) - age 6: 821056 bytes, 99544704 total
] GC(11438) - age 7: 2690424 bytes, 102235128 total
] GC(11438) - age 8: 691936 bytes, 102927064 total
] GC(11438) - age 9: 911824 bytes, 103838888 total
] GC(11438) - age 10: 3465976 bytes, 107304864 total
] GC(11438) - age 11: 5727008 bytes, 113031872 total
] GC(11438) - age 12: 5369776 bytes, 118401648 total
] GC(11438) - age 13: 674984 bytes, 119076632 total
] GC(11438) - age 14: 6494888 bytes, 125571520 total
] GC(11438) - age 15: 5776032 bytes, 131347552 total
] GC(11438) Eden regions: 1104->0(1108)
] GC(11438) Survivor regions: 22->18(141)
] GC(11438) Old regions: 175->176
] GC(11438) Humongous regions: 7->7
][gc,metaspace] GC(11438) Metaspace: 199667K(212928K)->199667K(212928K) NonClass: 173313K(184128K)->173313K(184128K) Class: 26354K(28800K)->26354K(28800K)
] GC(11438) Pause Young (Normal) (G1 Evacuation Pause) 10452M->1596M(15024M) 73.278ms
] GC(11438) User=0.54s Sys=0.01s Real=0.07s
] Safepoint "G1CollectForAllocation", Time since last: 74892495300 ns, Reaching safepoint: 13100 ns, Cleanup: 3500 ns, At safepoint: 73433400 ns, Total: 73450000 ns
] Safepoint "Cleanup", Time since last: 278020487500 ns, Reaching safepoint: 28113100 ns, Cleanup: 77600 ns, At safepoint: 94500 ns, Total: 28285200 ns
] Safepoint "Cleanup", Time since last: 61191135800 ns, Reaching safepoint: 10000 ns, Cleanup: 7600 ns, At safepoint: 11700 ns, Total: 29300 ns
] GC(11439) Pause Young (Normal) (G1 Evacuation Pause)
] GC(11439) Using 8 workers of 8 for evacuation
] GC(11439) Desired survivor size 591396864 bytes, new threshold 15 (max threshold 15)
] GC(11439) Pre Evacuate Collection Set: 0.6ms
] GC(11439) Merge Heap Roots: 0.4ms
] GC(11439) Evacuate Collection Set: 63.3ms
] GC(11439) Post Evacuate Collection Set: 10.9ms
] GC(11439) Other: 0.1ms
] GC(11439) Age table with threshold 15 (max threshold 15)
] GC(11439) - age 1: 86944384 bytes, 86944384 total
] GC(11439) - age 2: 13775480 bytes, 100719864 total
] GC(11439) - age 3: 8299480 bytes, 109019344 total
] GC(11439) - age 4: 3221864 bytes, 112241208 total
] GC(11439) - age 5: 5818960 bytes, 118060168 total
] GC(11439) - age 6: 4455608 bytes, 122515776 total
] GC(11439) - age 7: 797616 bytes, 123313392 total
] GC(11439) - age 8: 2690392 bytes, 126003784 total
] GC(11439) - age 9: 691936 bytes, 126695720 total
] GC(11439) - age 10: 911824 bytes, 127607544 total
] GC(11439) - age 11: 3464360 bytes, 131071904 total
] GC(11439) - age 12: 5726920 bytes, 136798824 total
] GC(11439) - age 13: 5360200 bytes, 142159024 total
] GC(11439) - age 14: 673656 bytes, 142832680 total
] GC(11439) - age 15: 6494888 bytes, 149327568 total
] GC(11439) Eden regions: 1108->0(1107)
] GC(11439) Survivor regions: 18->19(141)
] GC(11439) Old regions: 176->176
] GC(11439) Humongous regions: 7->7
][gc,metaspace] GC(11439) Metaspace: 199681K(212928K)->199681K(212928K) NonClass: 173325K(184128K)->173325K(184128K) Class: 26356K(28800K)->26356K(28800K)
] GC(11439) Pause Young (Normal) (G1 Evacuation Pause) 10460M->1614M(15024M) 75.434ms
] GC(11439) User=0.56s Sys=0.00s Real=0.07s
] Safepoint "G1CollectForAllocation", Time since last: 25232650200 ns, Reaching safepoint: 52700 ns, Cleanup: 3900 ns, At safepoint: 75591300 ns, Total: 75647900 ns
] Safepoint "Cleanup", Time since last: 313591218000 ns, Reaching safepoint: 14000 ns, Cleanup: 10300 ns, At safepoint: 3800 ns, Total: 28100 ns
] GC(11440) Pause Young (Normal) (G1 Evacuation Pause)
] GC(11440) Using 8 workers of 8 for evacuation
] GC(11440) Desired survivor size 591396864 bytes, new threshold 15 (max threshold 15)
] GC(11440) Pre Evacuate Collection Set: 0.8ms
] GC(11440) Merge Heap Roots: 1.1ms
] GC(11440) Evacuate Collection Set: 69.6ms
] GC(11440) Post Evacuate Collection Set: 9.1ms
] GC(11440) Other: 0.3ms
] GC(11440) Age table with threshold 15 (max threshold 15)
] GC(11440) - age 1: 88233288 bytes, 88233288 total
] GC(11440) - age 2: 14912960 bytes, 103146248 total
] GC(11440) - age 3: 13722584 bytes, 116868832 total
] GC(11440) - age 4: 8195296 bytes, 125064128 total
] GC(11440) - age 5: 3196680 bytes, 128260808 total
] GC(11440) - age 6: 5729944 bytes, 133990752 total
] GC(11440) - age 7: 4451656 bytes, 138442408 total
] GC(11440) - age 8: 796832 bytes, 139239240 total
] GC(11440) - age 9: 2690336 bytes, 141929576 total
] GC(11440) - age 10: 673928 bytes, 142603504 total
] GC(11440) - age 11: 911824 bytes, 143515328 total
] GC(11440) - age 12: 3446296 bytes, 146961624 total
] GC(11440) - age 13: 5726672 bytes, 152688296 total
] GC(11440) - age 14: 5341928 bytes, 158030224 total
] GC(11440) - age 15: 673656 bytes, 158703880 total
] GC(11440) Eden regions: 1107->0(1105)
] GC(11440) Survivor regions: 19->21(141)
] GC(11440) Old regions: 176->177
] GC(11440) Humongous regions: 7->7
][gc,metaspace] GC(11440) Metaspace: 199681K(212928K)->199681K(212928K) NonClass: 173325K(184128K)->173325K(184128K) Class: 26356K(28800K)->26356K(28800K)
] GC(11440) Pause Young (Normal) (G1 Evacuation Pause) 10470M->1632M(15024M) 81.205ms
] GC(11440) User=0.60s Sys=0.00s Real=0.08s
] Safepoint "G1CollectForAllocation", Time since last: 137830089000 ns, Reaching safepoint: 137800 ns, Cleanup: 8400 ns, At safepoint: 81359800 ns, Total: 81506000 ns
] Safepoint "Cleanup", Time since last: 256768954600 ns, Reaching safepoint: 16500 ns, Cleanup: 13600 ns, At safepoint: 10000 ns, Total: 40100 ns
] Safepoint "Cleanup", Time since last: 69936291800 ns, Reaching safepoint: 12700 ns, Cleanup: 7200 ns, At safepoint: 8300 ns, Total: 28200 ns
] GC(11441) Pause Young (Normal) (G1 Evacuation Pause)
] GC(11441) Using 8 workers of 8 for evacuation
] GC(11441) Desired survivor size 591396864 bytes, new threshold 15 (max threshold 15)
] GC(11441) Pre Evacuate Collection Set: 0.4ms
] GC(11441) Merge Heap Roots: 0.5ms
] GC(11441) Evacuate Collection Set: 59.7ms
] GC(11441) Post Evacuate Collection Set: 8.4ms
] GC(11441) Other: 0.1ms
] GC(11441) Age table with threshold 15 (max threshold 15)
] GC(11441) - age 1: 99309032 bytes, 99309032 total
] GC(11441) - age 2: 7187240 bytes, 106496272 total
] GC(11441) - age 3: 9993344 bytes, 116489616 total
] GC(11441) - age 4: 12977720 bytes, 129467336 total
] GC(11441) - age 5: 5768240 bytes, 135235576 total
] GC(11441) - age 6: 2725216 bytes, 137960792 total
] GC(11441) - age 7: 5033576 bytes, 142994368 total
] GC(11441) - age 8: 4260960 bytes, 147255328 total
] GC(11441) - age 9: 796832 bytes, 148052160 total
] GC(11441) - age 10: 2690016 bytes, 150742176 total
] GC(11441) - age 11: 673896 bytes, 151416072 total
] GC(11441) - age 12: 911808 bytes, 152327880 total
] GC(11441) - age 13: 3445896 bytes, 155773776 total
] GC(11441) - age 14: 5707496 bytes, 161481272 total
] GC(11441) - age 15: 5323608 bytes, 166804880 total
] GC(11441) Eden regions: 1105->0(1104)
] GC(11441) Survivor regions: 21->22(141)
] GC(11441) Old regions: 177->177
] GC(11441) Humongous regions: 7->7
][gc,metaspace] GC(11441) Metaspace: 199737K(212928K)->199737K(212928K) NonClass: 173371K(184128K)->173371K(184128K) Class: 26365K(28800K)->26365K(28800K)
] GC(11441) Pause Young (Normal) (G1 Evacuation Pause) 10472M->1643M(15024M) 69.410ms
] GC(11441) User=0.51s Sys=0.01s Real=0.08s
] Safepoint "G1CollectForAllocation", Time since last: 125455567600 ns, Reaching safepoint: 19900 ns, Cleanup: 4800 ns, At safepoint: 69565000 ns, Total: 69589700 ns
] Safepoint "Cleanup", Time since last: 169061884400 ns, Reaching safepoint: 13500 ns, Cleanup: 11600 ns, At safepoint: 4300 ns, Total: 29400 ns
] Safepoint "Cleanup", Time since last: 205250770600 ns, Reaching safepoint: 15100 ns, Cleanup: 15800 ns, At safepoint: 8700 ns, Total: 39600 ns
] GC(11442) Pause Young (Normal) (G1 Evacuation Pause)
] GC(11442) Using 8 workers of 8 for evacuation
] GC(11442) Desired survivor size 591396864 bytes, new threshold 15 (max threshold 15)
] GC(11442) Pre Evacuate Collection Set: 1.0ms
] GC(11442) Merge Heap Roots: 0.8ms
] GC(11442) Evacuate Collection Set: 80.3ms
] GC(11442) Post Evacuate Collection Set: 12.7ms
] GC(11442) Other: 0.2ms
] GC(11442) Age table with threshold 15 (max threshold 15)
] GC(11442) - age 1: 97894616 bytes, 97894616 total
] GC(11442) - age 2: 10630472 bytes, 108525088 total
] GC(11442) - age 3: 5030152 bytes, 113555240 total
] GC(11442) - age 4: 9321136 bytes, 122876376 total
] GC(11442) - age 5: 12375448 bytes, 135251824 total
] GC(11442) - age 6: 5140144 bytes, 140391968 total
] GC(11442) - age 7: 2723168 bytes, 143115136 total
] GC(11442) - age 8: 5012792 bytes, 148127928 total
] GC(11442) - age 9: 4260872 bytes, 152388800 total
] GC(11442) - age 10: 796832 bytes, 153185632 total
] GC(11442) - age 11: 2689744 bytes, 155875376 total
] GC(11442) - age 12: 673896 bytes, 156549272 total
] GC(11442) - age 13: 911728 bytes, 157461000 total
] GC(11442) - age 14: 3420040 bytes, 160881040 total
] GC(11442) - age 15: 5688256 bytes, 166569296 total
] GC(11442) Eden regions: 1104->0(1104)
] GC(11442) Survivor regions: 22->22(141)
] GC(11442) Old regions: 177->178
] GC(11442) Humongous regions: 7->7
][gc,metaspace] GC(11442) Metaspace: 199739K(212928K)->199739K(212928K) NonClass: 173374K(184128K)->173374K(184128K) Class: 26365K(28800K)->26365K(28800K)
] GC(11442) Pause Young (Normal) (G1 Evacuation Pause) 10475M->1648M(15024M) 95.402ms
] GC(11442) User=0.71s Sys=0.00s Real=0.10s
] Safepoint "G1CollectForAllocation", Time since last: 37152114400 ns, Reaching safepoint: 27900 ns, Cleanup: 6100 ns, At safepoint: 95559100 ns, Total: 95593100 ns
] Safepoint "Cleanup", Time since last: 59727583800 ns, Reaching safepoint: 16100 ns, Cleanup: 13100 ns, At safepoint: 8400 ns, Total: 37600 ns
] Safepoint "Cleanup", Time since last: 1202901000 ns, Reaching safepoint: 8500 ns, Cleanup: 8700 ns, At safepoint: 3600 ns, Total: 20800 ns

will there be any network or proxy level blocks to flow the data from the elastic agents to the cluster and we are getting the below error in the agent logs.
{"log.level":"error","@timestamp":"2024-01-30T12:43:02.159Z","message":"failed to perform any bulk index operations: Post "https://115.20.71.69:9200/_bulk?filter_path=errors%2Citems.*.error%2Citems.*.status\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"elasticsearch","log.origin":{"file.line":259,"file.name":"elasticsearch/client.go"},"ecs.version":"1.6.0"}

Christian_Dahlqvist · February 2, 2024, 12:40pm

That is given as dedicated master nodes can not hold indices. It does not answer my question though as I asked whether indexing requests are sent directly to the data nodes.

It sounds like you are actively indexing into all shards in the cluster, which can result in a lot of small and inefficient write requests as data in a single bulk request may go to a large number of shards.

Given that you are using SAN HDD storage I would recommend you check await and disk utilisation on the nodes, e.g. using iostat -x. As you likely are doing a lot of small writes due to your sharding it sould like this may very well be the bottleneck. If you look in the guidelines for optimising indexing performance you can see that the use of local SSDs (or storage with similar performance) is recommended.

Subrahmanyam_Veerank · February 5, 2024, 4:25am

Hi @Christian_Dahlqvist
The iostat result on node2 (data node) is below

,,,,,,
ELK02:~$ iostat -x
Linux 5.15.0-91-generic (ELK02) 02/05/2024 x86_64 (8 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
1.36 0.00 0.35 41.80 0.00 56.48

Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
dm-0 0.61 32.51 0.00 0.00 42.23 53.26 177.56 1305.22 0.00 0.00 10.95 7.35 0.01 621.35 0.00 0.00 0.33 109168.33 0.00 0.00 1.97 76.74
fd0 0.00 0.00 0.00 0.00 46.33 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop0 0.00 0.00 0.00 0.00 2.08 10.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop1 0.00 0.00 0.00 0.00 8.88 9.91 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop2 0.00 0.00 0.00 0.00 25.47 16.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop3 0.00 0.02 0.00 0.00 36.99 39.81 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop4 0.00 0.00 0.00 0.00 38.78 5.08 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.37 32.60 0.24 39.88 48.80 88.65 69.37 1305.28 108.34 60.97 9.77 18.82 0.01 622.33 0.00 11.93 0.35 123797.38 27.85 21.22 1.29 76.81
sr0 0.00 0.00 0.00 0.00 14.15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

,,,,,,,

Subrahmanyam_Veerank · February 5, 2024, 4:41am

Subrahmanyam_Veerank · February 5, 2024, 5:56am

yes and we havent used logstash also

Christian_Dahlqvist · February 5, 2024, 6:23am

What are you using to index into Elasticsearch? Are you using bulk requests? If so, what bulk request size are you using?

It sounds like you are not using time based indices and are updating/deleting data. I assume you are using external document IDs, is that correct? What is the use case?

How are you arranging your indices?

Based on the output you provided it seems like storage performance may very well be the bottleneck. I would recommend trying to switch to more performant storage and see if that makes a difference.

Subrahmanyam_Veerank · February 5, 2024, 6:57am

we are using nginx,kubernetes & nginx ingress controller integrations to ingest the logs of nginx error, access and container logs [enabled on agent policy with fleet] and output is given to elasticsearch. All the other things are kept as default only. we haven't modified anything. How to check bulk request size?
Yes, we were not updating/deleting data as of now.
All the policies/settings related to the indices are by default settings only.
But the %iowait on node3 and node4 is around 1.7-2 only.

Christian_Dahlqvist · February 5, 2024, 7:05am

I am not familiar with this so will need to leave that for someone else.

It looks like all indices are created with a single primary shard and that write load across indices are not evenly distributed. It is therefore possible that one of the nodes end up haing more shards for more heavily loaded indices and therefore becomes the bottleneck. It is possible for one node to be the bottleneck while other nodes are realtively less busy.

Subrahmanyam_Veerank · February 5, 2024, 7:08am

You can see that the indexing rate is too low and we were not able to find the exact issue...Kindly assign this to the concerned team for resolution.

Christian_Dahlqvist · February 5, 2024, 7:10am

This is a community forum where people volunteer their time. It is not a support forum, so there is no team to assign issues to. There are no SLAs or even guarantees you will get an answer/resolution.

I would recommend you follow the guidelines for optimising indexing performance and try to upgrade your storage.

Subrahmanyam_Veerank · February 5, 2024, 11:40am

Can anyone guide us to resolve this issue other than changing the storage?

Subrahmanyam_Veerank · February 6, 2024, 9:00am

Still we were not able to resolve this issue...Currently the indexing rate of overall cluster is about 400/s which is very low...Kindly support

Christian_Dahlqvist · February 6, 2024, 9:38am

You may want to verify that the issue is indeed with Elasticsearch and not in how you ingest data. You could create a new test index and index a large sample file into this directly using Logstah or Filebeat. If that shows the same low performance it is clearly an issue with the Elasticsearch cluster, but if that is able to index a lot faster the issue may be configuration or resources in the indexing pipeline.

system · March 5, 2024, 9:39am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Slow Indexing speed / Bottleneck Elasticsearch	6	721	September 16, 2020
Indexing rate performance in cluster Elasticsearch	6	3758	July 5, 2017
Low performance while searching Elasticsearch	5	500	November 27, 2020
Index mgmt as it relates to elastic ingest rate Elasticsearch	1	355	September 2, 2019
Elasticsearch Query performance while continuous ingestion Elasticsearch	5	576	August 15, 2019

Indexing rate became very low in elastic cluster

The indexing rate of our cluster become very low and even search results are getting delayed. cluster details:

Related topics

The indexing rate of our cluster become very low and even search results are getting delayed.
cluster details: