We have a cluster with 12 m4.4xlarge nodes (m4.4xlarge: 16vcores, 64GB memory). All the fields are keyword or long type. The cluster has about 4TB data, 24 indices, and 370 shards, 34.8 billions data. ES version is 5.5.2.
We found some aggregation search are very slow, which need about 15 seconds to return. These search are very ordinary which has one or two term filter and one term aggregations and one sum aggregation. Is this performance normal?
We checked the cluster stats, and found that doc_values_memory_in_bytes is very small. As we have about 32GB memory for OS on one node, why this value is so small. Does it mean that all the doc_values are in disk, not cache? Supposed it's big, will it be helpful to the performance? And how to tune it?
"docs" : {
"count" : 34823973828,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 4223269441404,
"throttle_time_in_millis" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 1756552,
"evictions" : 0
},
"query_cache" : {
"memory_size_in_bytes" : 30830935905,
"total_count" : 30419291,
"hit_count" : 21286292,
"miss_count" : 9132999,
"cache_size" : 8407,
"cache_count" : 8407,
"evictions" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 2290,
"memory_in_bytes" : 32573158916,
"terms_memory_in_bytes" : 29482647034,
"stored_fields_memory_in_bytes" : 1789047616,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 128,
"points_memory_in_bytes" : 1298264922,
"doc_values_memory_in_bytes" : 3199216,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 0,
"max_unsafe_auto_id_timestamp" : 1535587593485,
"file_sizes" : { }