How to check total required heap for ES 8.8

Elastic Stack Elasticsearch
1

To set my ES node's heap minimum, I referenced Size your shards (Size your shards | Elasticsearch Guide [8.8] | Elastic).

"total_deduplicated_mapping_size" : "4.1kb“
"total_estimated_overhead" : “13.5mb“
"extra heap for other overheads": 0.5GB
"master node" : 1GB

Finally I set '-Xms2g
-Xmx2g' in jvm.options.
then I restarted ES.
But, this ES Service go down due to OOM.

I think fielddata cache should be considerated.

How i can set Heap Memory size minimum without go down ?

2

From these docs:

0.5GB of extra heap will suffice for many reasonable workloads, and you may need even less if your workload is very light while heavy workloads may require more.

It's unfortunate that we cannot be more precise here, but it depends on so many other factors related to your workload. In your case, it sounds like you need to allow more than 0.5GB for your workload.

Why do you think the fielddata cache is the problem here?

3

I tested heap with fielddata cache.

  1. ES Service was loaded with heap 2g setting. but this was stopped due to OOM.
    I felt 2g is not enough.
    so I added more heap memory.

  2. ES Service was loaded with heap 3g setting. but this was also stopped due to OOM.
    I felt heap 3g is not enough too.

    heap3g
    heap3g865×569 95.8 KB

To check detail, I captured heap dump from ES. and I found fielddata do something wrong.

heapdump
heapdump1306×186 17.3 KB

  1. So I restricted fielddata size. ES Service was loaded with heap 3g, fielddata.cache.size:1GB setting. the service was go well.
    heap31g_fielddata.cache.size1GB
    heap31g_fielddata.cache.size1GB1253×718 54.8 KB

Finaly I realized "total_deduplicated_mapping_size","total_estimated_overhead","extra heap for other overheads" and "fielddata cache size" are considerated for heap size.

4

Thanks, that seems like a compelling analysis. However I'm puzzled because the fielddata circuit breaker should prevent this, limiting the size of this cache to 40% of the heap by default. Do you know why it didn't? For instance, what does GET /_nodes/_all/stats/breaker?filter_path=nodes.*.breakers.fielddata report?

5

I checked it with my last environments (ES Service with heap 3g, fielddata.cache.size:1GB setting).
"GET /_nodes/_all/stats/breaker?filter_path=nodes.*.breakers.fielddata" returned below

{
  "nodes": {
    "K6V95L0pR36L-_99LIapdw": {
      "breakers": {
        "fielddata": {
          "limit_size_in_bytes": 1288490188,
          "limit_size": "1.1gb",
          "estimated_size_in_bytes": 1047111152,
          "estimated_size": "998.6mb",
          "overhead": 1.03,
          "tripped": 0
        }
      }
    }
  }
}

PS. I used ES version 8.6 (not V8.8)

6

That makes sense, but what about in the case where you don't set fielddata.cache.size? The "limit_size": "1.1gb" should still apply.

Alternatively, the heap dump you captured showed 2.5GiB of heap being retained by the cache. Is that reflected in the circuit breaker and/or the stats (which you can compute from all the org.elasticsearch.index.fielddata.ShardFieldData#perFieldTotals maps)? Or is Elasticsearch not tracking some of this memory usage?

7

i retested ES Service with heap 3g setting (no setting fielddata.cache.size).
ES returned about fielddata below before stopping. Maybe Fielddata circuit breaker didn't control it.

{
  "nodes": {
    "K6V95L0pR36L-_99LIapdw": {
      "breakers": {
        "fielddata": {
          "limit_size_in_bytes": 1288490188,
          "limit_size": "1.1gb",
          "estimated_size_in_bytes": 2766456144,
          "estimated_size": "2.5gb",
          "overhead": 1.03,
          "tripped": 0
        }
      }
    }
  }
}
8

Thanks, that's helpful. I think this is a bug so I opened an issue in Github for you:

9

Thank you about fixing bug. but I need more.
As i said above, ES Service was loaded with heap 3g, fielddata.cache.size:1GB setting.
And when i add more index, heap is more required.. but, "total_deduplicated_mapping_size", "total_estimated_overhead" values are uppered little bit.
to check sufficient heap size, what i have to check more?

10

I don't have a good answer to this (at least nothing more specific than "your workload"). If you limit the fielddata cache to 1GiB, what else is consuming too much heap in your system?

11

I also use ES 5.6. the total number of org.elasticsearch.index.IndexService objects is equal to the number of open indexes In ES 5.6.
But the total number of org.elasticsearch.index.IndexService objects is equal to the number of all indexes In ES 8.6. why so many indexserves objects are loaded in the Service? Can i reduce the count for small heap size?

12

Can i reduce the count for small heap size?

No. The behaviour in 5.6 was a bug, long-since fixed.

13

In ES 5.6, if 10 indexes exist and 2 indexes open, 2 org.elasticsearch.index.IndexService objects loaded.

but, In ES 8.6, although 10 indexes exist and 2 indexes open, 10 org.elasticsearch.index.IndexService objects loaded.

if 10 indexes exist and 2 indexes open, can I load only 2 org.elasticsearch.index.IndexService objects In ES 8.6?

14

I'm not sure how this differs from your previous question. The behaviour in 5.6 that you describe was essentially due to a bug. There should be one IndexService for every index, open or closed, although the closed ones will be quite lightweight and won't load any field data.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.