JVM heap usage is high

Elastic Stack Elasticsearch
1

Hi,

We have 7 nodes cluster with 3 masters and 4 data nodes. We have one index with,
4 shards,
1 replica
size - 3.5GB
12894 documents
which is getting 100/s search requests normally and indexing rate is very low. But when peak hours reached index is hitting around 500 - 600 /s search requests. When that happens JVM heap is going up and hitting 99% and then data node is going down.

Search query :-
GET rxxxxxxxx_pxxxxxxx/_search
{
"size": 10000,
"query": {
"bool": {
"must": [
{
"term": {
"promotionDisplayStatus": {
"value": true,
"boost": 1
}
}
},
{
"terms": {
"conditionEntityFieldId": [6],
"boost": 1
}
},
{
"range": {
"promotionActiveUntil": {
"from": "2019-11-20",
"to": null,
"include_lower": true,
"include_upper": true,
"boost": 1
}
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
}
}

Could you please advice me to where or which setting needs to be tweaked?

2

Hi @chanakagl

to better help you, could you tell me which version of ES you're running?

3

version 6.8.2

4

Thanks @chanakagl

You could try tweaking the circuit breaker settings, see documentation on them here.

In particular a lower value for the in-flight circuit breaker or the parent circuit breaker might be helpful. You could also look into lowering the size of your search thread-pool, see docs on that [here] (https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-threadpool.html) from the defaults.

If that doesn't help increasing the heap size of your data node JVMs might be unavoidable to make 6.8.2 perform in this scenario. Another option though would be to upgrade to 7.x which has strong circuit breaker logic in place through the real-memory circuit breaker.

5

Thanks for quick response. I'll have a look on those areas. Meantime if i reduce no of shard to 1 and increase replica to 2 or 3 since index size only around 3.5GB with less documents, would that help?

6

@chanakagl

If your shard size is indeed that small, yes moving to a single shard may be a good optimisation (especially when the indexing rate is low).
You'd have to use 3 replicas though to make full use of your 4 data nodes if I understand your setup correctly (with 2 one of your nodes will just be unused).

7

Thanks i'll make those changes and see. Thank you very much again..!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.