High ram usage

[fnitz]

Hello,

after some posts and good answers we optimize our Elasticsearch.
Actually we use:
6 x hot nodes a 32 gb ram / heap space a 16 gb
22 x cold nodes a 16 gb / 8 gb space
We reduce 8 primaries shards to 6 shards

At the moment we see lower shards and less cpu usage. But we thought that less ram usage will be the result. But it raise day by day.

What can we do to reduce memory usage?

Thanks a lot.

[Christian_Dahlqvist]

Which version of Elasticsearch are you using? Are you forcemerging older read-only indices down to a single segment?

@fnitz

We reduce 8 primaries shards to 6 shards

How do you do this?

[fnitz]

  "name" : "kibana-com-1-rz1",
  "cluster_name" : "elastic-com-c1",
  "cluster_uuid" : "pk3BirT2SB-Z5MAPU5vC8A",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[fnitz]

We reduce the shards by chaging the index lifecycle policies

[Christian_Dahlqvist]

I would recommend you upgrade your cluster to version 8.7 as some significant improvement around heap usage have been made.

@fnitz

We reduce the shards by chaging the index lifecycle policies

Reducing the number of primary shards does not necessarily reduce heap usage a lot. It is, at least on the version you are on, important to also forcemerge down to a single segment.

[fnitz]

Thanks a lot.
Actually we see shards, heap space and segments are shrinking:

As I understood, with less segments we need less memory.
Besides the update we're planning, we didn't try to forcemerge to a single segment.

But which processes need memory besides the heap space?
I see in our dashboard that many metrics declining, but with no impacts on the entire memory usage....

[Christian_Dahlqvist]

Elasticsearch uses the heap, but also some off-heap memory. In addition to the the operating system page cache is essential for performance and it is common to see all memory in use on an Elasticsearch host.