# My cluster frequently has shards on initializing\_shards
**URL:** https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446
**Category:** Elasticsearch
**Created:** [January 18, 2023, 6:24pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446 "2023-01-18T18:24:12Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@Lohanna\_Sarah](https://discuss.elastic.co/u/Lohanna_Sarah)
#### Post date: [January 18, 2023, 6:24pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/1 "2023-01-18T18:24:12Z")
My cluster frequently has shards in the initializing\_shards state. The cluster is running on ks8. Exists any fix that I could do to fix it? The data nodes are restarted at least once a day
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 18, 2023, 10:03pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/2 "2023-01-18T22:03:34Z")
Welcome to our community! 😃
Why are the nodes restarted?
What do the Elasticsearch logs show?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 19, 2023, 6:35am UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/3 "2023-01-19T06:35:57Z")
What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/cluster-stats.html)?
Which version of Elasticsearch are you using?
Why are you restarting Elasticsearch so often?
Do you see shards initialize only after restarts or also at other times?
---
### Author:  [@Lohanna\_Sarah](https://discuss.elastic.co/u/Lohanna_Sarah)
#### Post date: [January 19, 2023, 1:05pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/4 "2023-01-19T13:05:37Z")
Sometimes deletes the pod during node scale down or node kills the pod because it is out of memory.
We are using the version 7.14.1.
The cluster stats:
{
"\_nodes": {
"total": 5,
"successful": 5,
"failed": 0
},
"cluster\_name": "diario-alertas",
"cluster\_uuid": "T3fKh3vLQJGMbf-OEFHJ-Q",
"timestamp": 1674133629765,
"status": "green",
"indices": {
"count": 4,
"shards": {
"total": 16,
"primaries": 8,
"replication": 1.0,
"index": {
"shards": {
"min": 2,
"max": 10,
"avg": 4.0
},
"primaries": {
"min": 1,
"max": 5,
"avg": 2.0
},
"replication": {
"min": 1.0,
"max": 1.0,
"avg": 1.0
}
}
},
"docs": {
"count": 426210025,
"deleted": 86704911
},
"store": {
"size\_in\_bytes": 4673224925138,
"total\_data\_set\_size\_in\_bytes": 4673224925138,
"reserved\_in\_bytes": 0
},
"fielddata": {
"memory\_size\_in\_bytes": 6686887732,
"evictions": 0
},
"query\_cache": {
"memory\_size\_in\_bytes": 0,
"total\_count": 0,
"hit\_count": 0,
"miss\_count": 0,
"cache\_size": 0,
"cache\_count": 0,
"evictions": 0
},
"completion": {
"size\_in\_bytes": 0
},
"segments": {
"count": 1187,
"memory\_in\_bytes": 4510592,
"terms\_memory\_in\_bytes": 2283760,
"stored\_fields\_memory\_in\_bytes": 1329688,
"term\_vectors\_memory\_in\_bytes": 0,
"norms\_memory\_in\_bytes": 232192,
"points\_memory\_in\_bytes": 0,
"doc\_values\_memory\_in\_bytes": 664952,
"index\_writer\_memory\_in\_bytes": 0,
"version\_map\_memory\_in\_bytes": 0,
"fixed\_bit\_set\_memory\_in\_bytes": 0,
"max\_unsafe\_auto\_id\_timestamp": 1654525198398,
"file\_sizes": {}
},
"mappings": {
"field\_types": [
{
"name": "boolean",
"count": 1,
"index\_count": 1,
"script\_count": 0
},
{
"name": "date",
"count": 2,
"index\_count": 1,
"script\_count": 0
},
{
"name": "float",
"count": 2,
"index\_count": 1,
"script\_count": 0
},
{
"name": "keyword",
"count": 17,
"index\_count": 1,
"script\_count": 0
},
{
"name": "long",
"count": 12,
"index\_count": 1,
"script\_count": 0
},
{
"name": "object",
"count": 8,
"index\_count": 1,
"script\_count": 0
},
{
"name": "percolator",
"count": 1,
"index\_count": 1,
"script\_count": 0
},
{
"name": "text",
"count": 16,
"index\_count": 1,
"script\_count": 0
}
],
"runtime\_field\_types":
},
"analysis": {
"char\_filter\_types": ,
"tokenizer\_types": ,
"filter\_types": ,
"analyzer\_types": ,
"built\_in\_char\_filters": ,
"built\_in\_tokenizers": ,
"built\_in\_filters": ,
"built\_in\_analyzers":
},
"versions": [
{
"version": "7.14.1",
"index\_count": 4,
"primary\_shard\_count": 8,
"total\_primary\_bytes": 2336580588500
}
]
},
"nodes": {
"count": {
"total": 5,
"coordinating\_only": 0,
"data": 3,
"data\_cold": 0,
"data\_content": 0,
"data\_frozen": 0,
"data\_hot": 0,
"data\_warm": 0,
"ingest": 0,
"master": 2,
"ml": 0,
"remote\_cluster\_client": 0,
"transform": 0,
"voting\_only": 0
},
"versions": [
"7.14.1"
],
"os": {
"available\_processors": 5,
"allocated\_processors": 5,
"names": [
{
"name": "Linux",
"count": 5
}
],
"pretty\_names": [
{
"pretty\_name": "CentOS Linux 8",
"count": 5
}
],
"architectures": [
{
"arch": "amd64",
"count": 5
}
],
"mem": {
"total\_in\_bytes": 81604378624,
"free\_in\_bytes": 984485888,
"used\_in\_bytes": 80619892736,
"free\_percent": 1,
"used\_percent": 99
}
},
"process": {
"cpu": {
"percent": 2
},
"open\_file\_descriptors": {
"min": 379,
"max": 415,
"avg": 393
}
},
"jvm": {
"max\_uptime\_in\_millis": 420614266,
"versions": [
{
"version": "16.0.2",
"vm\_name": "OpenJDK 64-Bit Server VM",
"vm\_version": "16.0.2+7",
"vm\_vendor": "Eclipse Foundation",
"bundled\_jdk": true,
"using\_bundled\_jdk": true,
"count": 5
}
],
"mem": {
"heap\_used\_in\_bytes": 19427339400,
"heap\_max\_in\_bytes": 60557361152
},
"threads": 190
},
"fs": {
"total\_in\_bytes": 9740860465152,
"free\_in\_bytes": 5050001469440,
"available\_in\_bytes": 5049917583360
},
"plugins": ,
"network\_types": {
"transport\_types": {
"security4": 5
},
"http\_types": {
"security4": 5
}
},
"discovery\_types": {
"zen": 5
},
"packaging\_types": [
{
"flavor": "default",
"type": "docker",
"count": 5
}
],
"ingest": {
"number\_of\_pipelines": 2,
"processor\_stats": {
"gsub": {
"count": 0,
"failed": 0,
"current": 0,
"time\_in\_millis": 0
},
"script": {
"count": 0,
"failed": 0,
"current": 0,
"time\_in\_millis": 0
}
}
}
}
}
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 19, 2023, 2:10pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/5 "2023-01-19T14:10:39Z")
> [@Lohanna\_Sarah](#):
>
> Sometimes deletes the pod during node scale down or node kills the pod because it is out of memory.
Adding or removing data nodes will always cause reallocation and rebalancing, so if this happens that is expected. I would generally recommend not autoscaling Elasticsearch for this reason.
> [@Lohanna\_Sarah](#):
>
> We are using the version 7.14.1.
This is quite old and I would recommend upgrading.
> [@Lohanna\_Sarah](#):
>
> "shards": {
> "total": 16,
> "primaries": 8,
> [@Lohanna\_Sarah](#):
>
> "store": {
> "size\_in\_bytes": 4673224925138,
It looks like you have very, very large shards (average of 292GB?), which will take time and resources to relocate. This likely means rebalancing will be slow and shards will take a long time to initialize. I would recommend increasing the number of primary shards in order to bring the shard size down to around 50GB or so.
> [@Lohanna\_Sarah](#):
>
> "master": 2,
Having just 2 master eligible nodes is very bad as a minimum or 3 master eligible nodes are required in order for the cluster to continue operating fully if one of the master eligible nodes fail or becomes unavailable. You should look to increase this to 3.
> [@Lohanna\_Sarah](#):
>
> "mem": {
> "heap\_used\_in\_bytes": 19427339400,
> "heap\_max\_in\_bytes": 60557361152
> [@Lohanna\_Sarah](#):
>
> "mem": {
> "total\_in\_bytes": 81604378624,
It looks like your heap is set to more than 50% of available RAM, which is not recommended. Elasticsearch uses off-heap memory and relies on the operating system cache for performance. Ensure you increase RAM to correct this. This could very well be why pods are running out of memory and get killed.
---
### Author:  [@Lohanna\_Sarah](https://discuss.elastic.co/u/Lohanna_Sarah)
#### Post date: [January 19, 2023, 4:27pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/6 "2023-01-19T16:27:05Z")
> [@Christian\_Dahlqvist](#):
>
> It looks like your heap is set to more than 50% of available RAM, which is not recommended. Elasticsearch uses off-heap memory and relies on the operating system cache for performance. Ensure you increase RAM to correct this. This could very well be why pods are running out of memory and get killed.
Is it possible to configure a limit to the heap memory?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 19, 2023, 4:35pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/7 "2023-01-19T16:35:20Z")
How are the nodes configured? How much resources are asigned?
---
### Author:  [@Lohanna\_Sarah](https://discuss.elastic.co/u/Lohanna_Sarah)
#### Post date: [January 19, 2023, 5:05pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/8 "2023-01-19T17:05:07Z")
> [@Christian\_Dahlqvist](#):
>
> How are the nodes configured? How much resources are asigned?
24gb of ram and the hard disk has 3T
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 19, 2023, 6:44pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/9 "2023-01-19T18:44:52Z")
Are all nodes the same specification? What is the heap size set to?
---
### Author:  [@Lohanna\_Sarah](https://discuss.elastic.co/u/Lohanna_Sarah)
#### Post date: [January 19, 2023, 7:44pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/10 "2023-01-19T19:44:46Z")
> [@Christian\_Dahlqvist](#):
>
> Are all nodes the same specification? What is the heap size set to?
Yes, using the cat API I had the following values:
heap.current heap.max name
6.5gb 18gb diario-alertas-es-data-nodes-2
257.5mb 1.1gb diario-alertas-es-master-nodes-0
509.4mb 1.1gb diario-alertas-es-master-nodes-1
8.4gb 18gb diario-alertas-es-data-nodes-0
966.6mb 18gb diario-alertas-es-data-nodes-1
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 19, 2023, 8:03pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/11 "2023-01-19T20:03:17Z")
So the nodes have 18GB heap on 24GB RAM? That should be no more than 50%, so i would recommend increasing RAM or reducing yhe heap size (assuming this does not lead to issues with GC).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 19, 2023, 9:54pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/12 "2023-01-19T21:54:08Z")
@Lohanna_Sarah just a note to please format your code/logs/config using the `` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂
---
### Author:  [@Lohanna\_Sarah](https://discuss.elastic.co/u/Lohanna_Sarah)
#### Post date: [January 19, 2023, 9:57pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/13 "2023-01-19T21:57:03Z")
> [@Christian\_Dahlqvist](#):
>
> So the nodes have 18GB heap on 24GB RAM? That should be no more than 50%, so i would recommend increasing RAM or reducing yhe heap size (assuming this does not lead to issues with GC).
Thanks, I will try to reduce the heap size.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 16, 2023, 9:58pm UTC](https://discuss.elastic.co/t/my-cluster-frequently-has-shards-on-initializing-shards/323446/14 "2023-02-16T21:58:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.