Hello.
I am curious about the increase in memory usage even though the query traffic is low and the number of documents is small, and I would like expert opinions on how to tune the GC.
[Servers (Nodes)]
- Total: 4 Nodes (master: 1, data: 3)
- CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (40 cores)
- MEM: 126GB
[ElasticSearch]
- Version: ES7.10.2 oss
- Number of Shards: 4,
- Number of Replicas: 1
- The number of indexed documents is about 20,000.
- 1 request per 1 second.
- 20,000 doucuments are updated only at 5am. (by batch)
- store.size: 57mb
- Not use Kibana, Logstash, Beats
** Scheme
*** about 80 fields (indexed fields: 30)
# Cluster API
{ -
"_nodes": { -
"total": 4,
"successful": 4,
"failed": 0
},
"cluster_name": "podcast1",
"cluster_uuid": "OIlUdP3JQRuGvUpwodvnIw",
"timestamp": 1623600113895,
"status": "green",
"indices": { -
"count": 1,
"shards": { -
"total": 8,
"primaries": 4,
"replication": 1,
"index": { -
"shards": { -
"min": 8,
"max": 8,
"avg": 8
},
"primaries": { -
"min": 4,
"max": 4,
"avg": 4
},
"replication": { -
"min": 1,
"max": 1,
"avg": 1
}
}
},
"docs": { -
"count": 21292,
"deleted": 0
},
"store": { -
"size_in_bytes": 60097104,
"reserved_in_bytes": 0
},
"fielddata": { -
"memory_size_in_bytes": 0,
"evictions": 0
},
"query_cache": { -
"memory_size_in_bytes": 0,
"total_count": 0,
"hit_count": 0,
"miss_count": 0,
"cache_size": 0,
"cache_count": 0,
"evictions": 0
},
"completion": { -
"size_in_bytes": 0
},
"segments": { -
"count": 8,
"memory_in_bytes": 128480,
"terms_memory_in_bytes": 96384,
"stored_fields_memory_in_bytes": 4160,
"term_vectors_memory_in_bytes": 0,
"norms_memory_in_bytes": 17536,
"points_memory_in_bytes": 0,
"doc_values_memory_in_bytes": 10400,
"index_writer_memory_in_bytes": 0,
"version_map_memory_in_bytes": 0,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": -1,
"file_sizes": { -
}
},
"mappings": { -
"field_types": [ -
{ -
"name": "date",
"count": 1,
"index_count": 1
},
{ -
"name": "integer",
"count": 11,
"index_count": 1
},
{ -
"name": "keyword",
"count": 21,
"index_count": 1
},
{ -
"name": "long",
"count": 8,
"index_count": 1
},
{ -
"name": "text",
"count": 39,
"index_count": 1
}
]
},
"analysis": { -
"char_filter_types": [ -
],
"tokenizer_types": [ -
{ -
"name": "pattern",
"count": 1,
"index_count": 1
}
],
"filter_types": [ -
{ -
"name": "pattern_replace",
"count": 1,
"index_count": 1
}
],
"analyzer_types": [ -
{ -
"name": "custom",
"count": 2,
"index_count": 1
}
],
"built_in_char_filters": [ -
],
"built_in_tokenizers": [ -
{ -
"name": "keyword",
"count": 1,
"index_count": 1
}
],
"built_in_filters": [ -
{ -
"name": "lowercase",
"count": 2,
"index_count": 1
}
],
"built_in_analyzers": [ -
]
}
},
"nodes": { -
"count": { -
"total": 4,
"coordinating_only": 0,
"data": 4,
"ingest": 4,
"master": 4,
"remote_cluster_client": 4
},
"versions": [ -
"7.10.2"
],
"os": { -
"available_processors": 160,
"allocated_processors": 160,
"names": [ -
{ -
"name": "Linux",
"count": 4
}
],
"pretty_names": [ -
{ -
"pretty_name": "CentOS Linux 7 (Core)",
"count": 4
}
],
"mem": { -
"total_in_bytes": 269138096128,
"free_in_bytes": 44883550208,
"used_in_bytes": 224254545920,
"free_percent": 17,
"used_percent": 83
}
},
"process": { -
"cpu": { -
"percent": 0
},
"open_file_descriptors": { -
"min": 360,
"max": 368,
"avg": 363
}
},
"jvm": { -
"max_uptime_in_millis": 14920332,
"versions": [ -
{ -
"version": "15.0.1",
"vm_name": "OpenJDK 64-Bit Server VM",
"vm_version": "15.0.1+9",
"vm_vendor": "AdoptOpenJDK",
"bundled_jdk": true,
"using_bundled_jdk": true,
"count": 4
}
],
"mem": { -
"heap_used_in_bytes": 9423393920,
"heap_max_in_bytes": 68719476736
},
"threads": 661
},
"fs": { -
"total_in_bytes": 2397027188736,
"free_in_bytes": 1984199778304,
"available_in_bytes": 1984199778304
},
"plugins": [ -
{ -
"name": "es-klp-plugin", # Korean Analyzer
"version": "1.12.5",
"elasticsearch_version": "7.10.2",
"java_version": "1.8",
"description": "SKT klp - ES Plugin",
"classname": "search.tech.plugin.KlpPlugin",
"extended_plugins": [ -
],
"has_native_controller": false
}
],
"network_types": { -
"transport_types": { -
"netty4": 4
},
"http_types": { -
"netty4": 4
}
},
"discovery_types": { -
"zen": 4
},
"packaging_types": [ -
{ -
"flavor": "oss",
"type": "tar",
"count": 4
}
],
"ingest": { -
"number_of_pipelines": 0,
"processor_stats": { -
}
}
}
}
[JDK and JVM options]
- Open JDK 15 (Bundled OpenJDK)
- Use G1GC
- Heap:
# I didn't change anything other than the heap size.
-Xms16g -Xmx16g
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30
Query requests are about 1 per second and there are almost no updates of documents.
I don't understand why the memory goes up to 60%.
(And at 5am, heap memory usage increases about 10% point by batch of updating 20000 documents)
The increase to 60% seems to be because the default value of the "-XX:G1MaxNewSizePercent" option is 60%.
I think it is a good situation that the heap memory graph is a saw feature,
But I hope that the slope of the graph decreases, resulting in a gradual memory rise and a short gc.
What should I do?