Elasticsearch Memory Consumption

Elastic Stack Elasticsearch
1

Hello,

In the current system we have 90 days of logs. However, most of the time, only 30 days of data is queried. So, I am thinking of following optimizations to manage old indices to reduce system memory utilization.

Option 1)

  1. Moving all primary shards to one node
  2. Shrink the data
  3. Forcemerge the data to a single segment (max documents is 120 million and single segment size is about 35 GB)

Will this help save heap memory significantly? How will this impact the query performance?

My current heap and ram usage is very high. Shown Below . Here ram.percent is quite high. However, sufficient RAM is free, about 20 GB of 64 GB. Why is this ram.percent very high?

curl -X GET "http://IP:PORT/_cat/nodes?v"
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
HOST1 86 98 4 1.25 1.23 1.25 mdi - node-3
HOST2 93 97 6 1.52 1.44 1.45 mdi - node-1
HOST3 92 98 4 0.88 1.26 1.31 mdi * node-2
HOST4 95 98 6 1.39 1.74 1.67 mdi - node-4

Option 2)

Close the old indices(older than 30 days). I can provide a mechanism to Open the indices when any closed index is queried and close the same after the query execution. Will this help save the system memory resources.

Which of the above options will help optimize the system resource memory usage and what is the recommended approach?

Index Stats for one of the node is given below :
{
"_nodes" : {
"total" : 4,
"successful" : 4,
"failed" : 0
},
"cluster_name" : "xxx-cluster",
"nodes" : {
"RQ50cjGaTJyDflmJqQKj5w" : {
"timestamp" : 1581074219257,
"name" : "node-3",
"transport_address" : "",
"host" : "",
"ip" : "",
"roles" : [
"master",
"data",
"ingest"
],
"attributes" : {
"ml.machine_memory" : "67378851840",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
},
"indices" : {
"docs" : {
"count" : 5088498671,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 1647950022235
},
"indexing" : {
"index_total" : 44423906,
"index_time_in_millis" : 18482112,
"index_current" : 0,
"index_failed" : 0,
"delete_total" : 0,
"delete_time_in_millis" : 0,
"delete_current" : 0,
"noop_update_total" : 0,
"is_throttled" : false,
"throttle_time_in_millis" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 10,
"query_total" : 14334,
"query_time_in_millis" : 530529,
"query_current" : 10,
"fetch_total" : 2864,
"fetch_time_in_millis" : 4748083,
"fetch_current" : 0,
"scroll_total" : 10,
"scroll_time_in_millis" : 47487324,
"scroll_current" : 0,
"suggest_total" : 0,
"suggest_time_in_millis" : 0,
"suggest_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 4507,
"total_time_in_millis" : 12197240,
"total_docs" : 171012154,
"total_size_in_bytes" : 56604653805,
"total_stopped_time_in_millis" : 0,
"total_throttled_time_in_millis" : 5493618,
"total_auto_throttle_in_bytes" : 17097031680
},
"refresh" : {
"total" : 15005,
"total_time_in_millis" : 3205850,
"listeners" : 0
},
"flush" : {
"total" : 539,
"periodic" : 76,
"total_time_in_millis" : 305456
},
"warmer" : {
"current" : 0,
"total" : 13505,
"total_time_in_millis" : 8185595
},
"query_cache" : {
"memory_size_in_bytes" : 38048113,
"total_count" : 76546,
"hit_count" : 72281,
"miss_count" : 4265,
"cache_size" : 106,
"cache_count" : 132,
"evictions" : 26
},
"fielddata" : {
"memory_size_in_bytes" : 15439073464,
"evictions" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 10578,
"memory_in_bytes" : 5687338579,
"terms_memory_in_bytes" : 5384658391,
"stored_fields_memory_in_bytes" : 239039736,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 1299520,
"points_memory_in_bytes" : 59617508,
"doc_values_memory_in_bytes" : 2723424,
"index_writer_memory_in_bytes" : 30111104,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 636700504,
"max_unsafe_auto_id_timestamp" : 1581019261329,
"file_sizes" : { }
},
"translog" : {
"operations" : 2230958,
"size_in_bytes" : 1385529357,
"uncommitted_operations" : 1316853,
"uncommitted_size_in_bytes" : 579925452,
"earliest_last_modified_age" : 0
},
"request_cache" : {
"memory_size_in_bytes" : 104000,
"evictions" : 0,
"hit_count" : 3,
"miss_count" : 25
},
"recovery" : {
"current_as_source" : 0,
"current_as_target" : 0,
"throttle_time_in_millis" : 2252275
}
}
},

2

Hi @guruprasadb

First off, I think Frozen Indices may be what you are looking for if you can live with slow queries on old data (opening an index just for querying would certainly be slower in most cases still so I'm assuming this option would work for you).

I don't think you'll be able to move all primaries to one node if you also have replicas. There is no way of doing that (and it's not really a useful thing to do, what if that node goes down ... then replicas will take over and your primaries are all over the place again).

  1. Forcemerge the data to a single segment (max documents is 120 million and single segment size is about 35 GB)

If the data isn't going to change any more this will save some memory and improve performance most likely I would assume. The exact degree to which this happens is hard to predict and something to try out I think.

Afaik ram.percentage includes page cache use as well, that's probably why the number is so high overall as well.

=> you could try force merging things that will never change again and see if that's enough.
=> Option 2) doesn't seem reasonable to me, when frozen indices are available. Opening an index is costly resource wise and not something that should be part of a one time query.

3

Dear Armin,

Thanks for your inputs. How can i improve the memory utilization given the below data. I am looking at optimizations other than freezing the old unused indices?

I keep running into frequent GC taking more than 30 seconds and any executing aggregation queries on many days of data times out and fails.

I have four nodes with 24 cores and 64GB RAM and 32G of JVM HEAP. Daily index and each index has about 100 million records. 90 days of data is retained in the system. Best Compression is used with 5 Shards and 1 replica.

Heap Utilization Percentage is more than 85%
Fielddata Memory is about 15 GB (per node)
Segment Memory is about 5.3 GB (per node)
Translog Size in Byttes is about 1.4 GB (per node)
Size of Compressed data consumed by all shards : 6511 GB (total size handled by 4 nodes. Sum of all values from output of "_cat/shards?")
"active_primary_shards" : 1645 (all nodes)

================================================================
Logs and output of different commands is given below

Output of : _nodes/stats/indices

"query_cache" : {
"memory_size_in_bytes" : 250483404,
"total_count" : 345506,
"hit_count" : 265900,
"miss_count" : 79606,
"cache_size" : 885,
"cache_count" : 2055,
"evictions" : 1170
},
"fielddata" : {
"memory_size_in_bytes" : 15607420544,
"evictions" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 10786,
"memory_in_bytes" : 5743504096,
"terms_memory_in_bytes" : 5435352021,
"stored_fields_memory_in_bytes" : 242611112,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 1295936,
"points_memory_in_bytes" : 61446083,
"doc_values_memory_in_bytes" : 2798944,
"index_writer_memory_in_bytes" : 17527760,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 648466280,
"max_unsafe_auto_id_timestamp" : 1581278460804,
"file_sizes" : { }
},
"translog" : {
"operations" : 2831157,
"size_in_bytes" : 1549152668,
"uncommitted_operations" : 1788929,
"uncommitted_size_in_bytes" : 609003907,
"earliest_last_modified_age" : 0
},
"request_cache" : {
"memory_size_in_bytes" : 44842106,
"evictions" : 0,
"hit_count" : 309,
"miss_count" : 8221
},

Output of : _cat/nodes?v

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
x.x.x.x 93 95 8 1.35 1.95 1.93 mdi - node-3
x.x.x.x 87 94 10 1.67 1.90 2.24 mdi - node-1
x.x.x.x 89 97 6 0.43 1.00 1.32 mdi * node-2
x.x.x.x 88 94 5 0.69 0.92 1.12 mdi - node-4

Output of : _cluster/stats?

"cluster_name" : "my-cluster",
"cluster_uuid" : "ntFmtDanQx2yTaivjvSPBw",
"timestamp" : 1581324608473,
"status" : "green",
"indices" : {
"count" : 329,
"shards" : {
"total" : 3290,
"primaries" : 1645,
"replication" : 1.0,
"index" : {
"shards" : {
"min" : 10,
"max" : 10,
"avg" : 10.0
},
"primaries" : {
"min" : 5,
"max" : 5,
"avg" : 5.0
},
"replication" : {
"min" : 1.0,
"max" : 1.0,
"avg" : 1.0
}
}
},
"docs" : {
"count" : 10358677517,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 6704778880187
},
"fielddata" : {
"memory_size_in_bytes" : 62723570200,
"evictions" : 0
},
"query_cache" : {
"memory_size_in_bytes" : 1195879730,
"total_count" : 1452057,
"hit_count" : 1080694,
"miss_count" : 371363,
"cache_size" : 5393,
"cache_count" : 10727,
"evictions" : 5334
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 42867,
"memory_in_bytes" : 23026343590,
"terms_memory_in_bytes" : 21794765416,
"stored_fields_memory_in_bytes" : 970973792,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 5217344,
"points_memory_in_bytes" : 244421530,
"doc_values_memory_in_bytes" : 10965508,
"index_writer_memory_in_bytes" : 279531472,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 2592254192,
"max_unsafe_auto_id_timestamp" : 1581278460804,
"file_sizes" : { }
}
},
"nodes" : {
"count" : {
"total" : 4,
"data" : 4,
"coordinating_only" : 0,
"master" : 4,
"ingest" : 4
},
"versions" : [
"6.6.1"
],
"os" : {
"available_processors" : 96,
"allocated_processors" : 96,
"names" : [
{
"name" : "Linux",
"count" : 4
}
],
"pretty_names" : [
{
"pretty_name" : "Red Hat Enterprise Linux",
"count" : 4
}
],
"mem" : {
"total_in_bytes" : 269515407360,
"free_in_bytes" : 14183038976,
"used_in_bytes" : 255332368384,
"free_percent" : 5,
"used_percent" : 95
}
},
"process" : {
"cpu" : {
"percent" : 19
},
"open_file_descriptors" : {
"min" : 11269,
"max" : 15789,
"avg" : 13896
}
},
"jvm" : {
"max_uptime_in_millis" : 564315186,
"versions" : [
{
"version" : "1.8.0_211",
"vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
"vm_version" : "25.211-b12",
"vm_vendor" : "Oracle Corporation",
"count" : 4
}
],
"mem" : {
"heap_used_in_bytes" : 119341024336,
"heap_max_in_bytes" : 137438953472
},
"threads" : 978
},
"fs" : {
"total_in_bytes" : 50306197454848,
"free_in_bytes" : 39109396488192,
"available_in_bytes" : 36553692176384
},

4

Hey @guruprasadb

could you please format that JSON in ``` for easier readability. Also, it would be nice if you made those API requests with the ?human flag set, so that all the numbers are a little easier to digest. Happy to take a look at them then :slight_smile:

Thanks!

5

The first thing I would recommend is to make your heap a bit smaller. I believe it should be set to no larger than around 30GB or so in order to benefit from compressed pointers, and this should allow you to use your heap more efficiently. I would also recommend forcemerging older indices that are no longer written to as this can save a lot of heap. Please see this webinar for further details.

6

Hi @Armin_Braun,
Pls find attached logs in the requested format

Output of "_cluster/stats?human&pretty"

```
{
  "_nodes" : {
    "total" : 4,
    "successful" : 4,
    "failed" : 0
  },
  "cluster_name" : "kca-cluster",
  "cluster_uuid" : "ntFmtDanQx2yTaivjvSPBw",
  "timestamp" : 1581396654650,
  "status" : "green",
  "indices" : {
    "count" : 332,
    "shards" : {
      "total" : 3320,
      "primaries" : 1660,
      "replication" : 1.0,
      "index" : {
        "shards" : {
          "min" : 10,
          "max" : 10,
          "avg" : 10.0
        },
        "primaries" : {
          "min" : 5,
          "max" : 5,
          "avg" : 5.0
        },
        "replication" : {
          "min" : 1.0,
          "max" : 1.0,
          "avg" : 1.0
        }
      }
    },
    "docs" : {
      "count" : 10434559500,
      "deleted" : 0
    },
    "store" : {
      "size" : "6.1tb",
      "size_in_bytes" : 6753453428508
    },
    "fielddata" : {
      "memory_size" : "58.8gb",
      "memory_size_in_bytes" : 63140716128,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size" : "1gb",
      "memory_size_in_bytes" : 1108472389,
      "total_count" : 2621091,
      "hit_count" : 1871281,
      "miss_count" : 749810,
      "cache_size" : 5002,
      "cache_count" : 12011,
      "evictions" : 7009
    },
    "completion" : {
      "size" : "0b",
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 43207,
      "memory" : "21.5gb",
      "memory_in_bytes" : 23179911513,
      "terms_memory" : "20.4gb",
      "terms_memory_in_bytes" : 21939195688,
      "stored_fields_memory" : "932.8mb",
      "stored_fields_memory_in_bytes" : 978138000,
      "term_vectors_memory" : "0b",
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory" : "5mb",
      "norms_memory_in_bytes" : 5256512,
      "points_memory" : "234.7mb",
      "points_memory_in_bytes" : 246197805,
      "doc_values_memory" : "10.6mb",
      "doc_values_memory_in_bytes" : 11123508,
      "index_writer_memory" : "263.5mb",
      "index_writer_memory_in_bytes" : 276315584,
      "version_map_memory" : "0b",
      "version_map_memory_in_bytes" : 0,
      "fixed_bit_set" : "2.4gb",
      "fixed_bit_set_memory_in_bytes" : 2611245704,
      "max_unsafe_auto_id_timestamp" : 1581374361483,
      "file_sizes" : { }
    }
  },
  "nodes" : {
    "count" : {
      "total" : 4,
      "data" : 4,
      "coordinating_only" : 0,
      "master" : 4,
      "ingest" : 4
    },
    "versions" : [
      "6.6.1"
    ],
    "os" : {
      "available_processors" : 96,
      "allocated_processors" : 96,
      "names" : [
        {
          "name" : "Linux",
          "count" : 4
        }
      ],
      "pretty_names" : [
        {
          "pretty_name" : "Red Hat Enterprise Linux",
          "count" : 4
        }
      ],
      "mem" : {
        "total" : "251gb",
        "total_in_bytes" : 269515407360,
        "free" : "6.5gb",
        "free_in_bytes" : 7033028608,
        "used" : "244.4gb",
        "used_in_bytes" : 262482378752,
        "free_percent" : 3,
        "used_percent" : 97
      }
    },
    "process" : {
      "cpu" : {
        "percent" : 14
      },
      "open_file_descriptors" : {
        "min" : 11318,
        "max" : 16368,
        "avg" : 14139
      }
    },
    "jvm" : {
      "max_uptime" : "7.3d",
      "max_uptime_in_millis" : 636359888,
      "versions" : [
        {
          "version" : "1.8.0_211",
          "vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
          "vm_version" : "25.211-b12",
          "vm_vendor" : "Oracle Corporation",
          "count" : 4
        }
      ],
      "mem" : {
        "heap_used" : "111.1gb",
        "heap_used_in_bytes" : 119387408584,
        "heap_max" : "128gb",
        "heap_max_in_bytes" : 137438953472
      },
      "threads" : 974
    },
    "fs" : {
      "total" : "45.7tb",
      "total_in_bytes" : 50306197454848,
      "free" : "35.4tb",
      "free_in_bytes" : 38951136686080,
      "available" : "33.1tb",
      "available_in_bytes" : 36395432374272
    },
```

Output of "_cat/nodes?human&v""

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.137.72.19 91 98 5 0.69 0.82 1.01 mdi - node-3
10.137.72.17 91 97 4 0.21 0.69 0.92 mdi - node-1
10.137.72.18 92 98 4 0.59 0.78 1.04 mdi * node-2
10.137.72.10 81 96 6 0.88 1.21 1.28 mdi - node-4

Output of "_nodes/stats/indices?human&pretty"

```
     "query_cache" : {
          "memory_size" : "250.8mb",
          "memory_size_in_bytes" : 263086511,
          "total_count" : 783703,
          "hit_count" : 560697,
          "miss_count" : 223006,
          "cache_size" : 948,
          "cache_count" : 2961,
          "evictions" : 2013
        },
        "fielddata" : {
          "memory_size" : "14.6gb",
          "memory_size_in_bytes" : 15712232712,
          "evictions" : 0
        },
        "completion" : {
          "size" : "0b",
          "size_in_bytes" : 0
        },
        "segments" : {
          "count" : 10858,
          "memory" : "5.3gb",
          "memory_in_bytes" : 5782855402,
          "terms_memory" : "5gb",
          "terms_memory_in_bytes" : 5472597131,
          "stored_fields_memory" : "232.9mb",
          "stored_fields_memory_in_bytes" : 244258304,
          "term_vectors_memory" : "0b",
          "term_vectors_memory_in_bytes" : 0,
          "norms_memory" : "1.2mb",
          "norms_memory_in_bytes" : 1305344,
          "points_memory" : "58.9mb",
          "points_memory_in_bytes" : 61864839,
          "doc_values_memory" : "2.6mb",
          "doc_values_memory_in_bytes" : 2829784,
          "index_writer_memory" : "92.6mb",
          "index_writer_memory_in_bytes" : 97185564,
          "version_map_memory" : "0b",
          "version_map_memory_in_bytes" : 0,
          "fixed_bit_set" : "622.6mb",
          "fixed_bit_set_memory_in_bytes" : 652867000,
          "max_unsafe_auto_id_timestamp" : 1581372833159,
          "file_sizes" : { }
        },
        "translog" : {
          "operations" : 5364685,
          "size" : "3.2gb",
          "size_in_bytes" : 3525966678,
          "uncommitted_operations" : 1710702,
          "uncommitted_size" : "878.8mb",
          "uncommitted_size_in_bytes" : 921504154,
          "earliest_last_modified_age" : 0
        },
        "request_cache" : {
          "memory_size" : "103mb",
          "memory_size_in_bytes" : 108006529,
          "evictions" : 0,
          "hit_count" : 779,
          "miss_count" : 26761
        },
```
7

Thank you for the reply @ [Christian_Dahlqvist]. I have some more queries.

  1. Is it recommended to use both Freeze Index and Force Merging the Segments to a Single Segment. Will Freeze index have any additional benefits?

  2. There is about 2TB of data with 3.5 Billion Documents for querying with 30 days of data. The cardinality of some of the Terms is in the order of 10 million. We have queries involving nested aggregation queries with 3-4 levels of nesting. Considering these, how many HOT nodes are recommended and typical sizing needed for these nodes. [I have shared the current system statistics]

8

I don’t think anyone can answer that. You need to test and benchmark.

Freezing indices that are rarely queried will reduce heap usage at the expense of query performance.

9

As @Christian_Dahlqvist points out, the concrete performance and resource needs you'll have to experiment out yourself, we can't guess/estimate them easily from afar, there's just too many variables playing into this.

Yes, as posted above you will save significantly on memory e.g. the following

will not be loaded on heap constantly or frozen indices. This will however also significantly reduce query performance but as I said may be a valid tradeoff in your use case.

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.