### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 8:53am UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/1 "2020-10-19T08:53:21Z")
Hi Team,
initializing\_shards got struck, if i delete those shards wat will happen? it will become Green and will work smoothly?
Kibana link is working for some time and its showing error timeout 30000ms some time.
{
"cluster\_name" : "OConnectElasticSearch",
**"status" : "red",**
"timed\_out" : false,
"number\_of\_nodes" : 9,
"number\_of\_data\_nodes" : 6,
"active\_primary\_shards" : 20474,
"active\_shards" : 40948,
"relocating\_shards" : 0,
**"initializing\_shards" : 4,**
"unassigned\_shards" : 4,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 0,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 0,
**"active\_shards\_percent\_as\_number" : 99.98046684246509**
}
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [October 19, 2020, 9:05am UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/2 "2020-10-19T09:05:38Z")
41000 shards on 6 nodes.... Which means around 6900 shards per node.
You probably have too many shards per node.
May I suggest you look at the following resources about sizing:
> **[Quantitative Cluster Sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)**
> **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**
> **[Managing your black Friday logs - CloudConf.IT](https://www.slideshare.net/dadoonet/managing-your-black-friday-logs-cloudconfit-93309899)**
>
> Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your a…
[ ](https://www.youtube.com/watch?v=ilP7tG6tabI)
And [Using Rally to Get Your Elasticsearch Cluster Size Right | Elastic Videos](https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right)
---
### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 9:19am UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/3 "2020-10-19T09:19:07Z")
Hi David,
Greetings !!
Thanks for your reply. I got you but for work around i need to fix this, because it is production one.
And also i am working on ELK upgradation. So very soon i will create new PROD environment there i will implement your suggestion. Here i need to fix ASAP as i said its in Production.
If i delete that 4 initializing shards what could happen ?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [October 19, 2020, 9:37am UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/4 "2020-10-19T09:37:49Z")
What is the output of:
```auto
GET /
GET /_cat/indices?v
```
If some outputs are too big, please share them on [gist.github.com](http://gist.github.com) and link them here.
> [@muthug](#):
>
> If i delete that 4 initializing shards what could happen ?
It will stay RED. As shards will be missing.
Best guess: wait for the cluster to recover.
Or Delete the indices which are RED but you will be missing some data.
---
### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 10:05am UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/5 "2020-10-19T10:05:23Z")
> [@dadoonet](#):
>
> ```auto
> GET /
> GET /_cat/indices?v
>
> ```
GET /
{
"name" : "OConnectManagementNode",
"cluster\_name" : "OConnectElasticSearch",
"version" : {
"number" : "2.3.4",
"build\_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
"build\_timestamp" : "2016-06-30T11:24:31Z",
"build\_snapshot" : false,
"lucene\_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}
GET /\_cat/indices?v
>
---
### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 10:12am UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/6 "2020-10-19T10:12:00Z")
> cat/shards |grep INITIALIZING
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0
vdc\_b\_e029-2020.04.12 5 p INITIALIZING OConnectDataNode04
vdc\_b\_err-2019.10.03 1 p INITIALIZING OConnectDataNode06
vdc\_b\_e029-2019.04.10 4 p INITIALIZING OConnectDataNode04
vdc\_b\_err-2020.09.18 1 p INITIALIZING OConnectDataNode04
100 3959k 100 3959k 0 0 453k 0 0:00:08 0:00:08 --:--:-- 837k
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 19, 2020, 10:23am UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/7 "2020-10-19T10:23:01Z")
> [@muthug](#):
>
> ```auto
> green open vdc_b_e058-2020.04.30 6 1 36 0 935.2kb 467.6kb
> green open vdc_b_e067-2020.09.28 6 1 75 0 925.3kb 462.6kb
> green open vdc_b_e067-2020.09.29 6 1 52 0 1.3mb 666.3kb
> green open vdc_b_e067-2020.09.10 6 1 11 0 289.1kb 144.5kb
> green open vdc_b_e067-2020.09.11 6 1 47 0 996.2kb 498.1kb
> green open vdc_b_e063-2019.06.01 6 1 77 0 719.6kb 359.8kb
>
> ```
Why would you have indices with less that 100 documents each and 6 primary shards and 6 replica shards. This is incredibly wasteful but explains why your cluster is so exceptionally oversharded. I would recommend you start addressing this ASAP (go to a single primary shard in your index templates, consolidate indices, switch from daily to e.g. monthly indices where the size is small) as it is just otherwise going to complicate the migration.
---
### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 10:30am UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/8 "2020-10-19T10:30:17Z")
True!! I am working on that. Soon it will be addressed and moved to new ELK stack 7.9.
Here what is the workaround solution to fix this?
If i don't want those shards data, shall i go for delete ? after that it would be fine or still remains in RED?
Note;- Kibana link is working but not consistently. Most of the time getting 30000ms timeout error.
I want to Fix Initializing\_shards and kibana timeout issue at-least for workaround ☹
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 19, 2020, 11:03am UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/9 "2020-10-19T11:03:11Z")
I would not be surprised if the Kibana timeout error originated from querying too many small shards.
---
### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 12:02pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/10 "2020-10-19T12:02:12Z")
So, You mean this timeout error is not because of these Initializing\_shards got struck, It because of too many small shards.
Correct me if i am wrong
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 19, 2020, 12:36pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/11 "2020-10-19T12:36:06Z")
It could be either. Querying large amounts of shards can be slow. Did you see timeouts before you has unallocated primary shards?
If you want to allocate the missing primary shards as empty shards you can use the [cluster reroute API](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/cluster-reroute.html), but be aware you will lose the data in those shards.
---
### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 1:19pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/12 "2020-10-19T13:19:58Z")
I have deleted the all 4 shards and the status turn into green. But Reallocating shards are showing 6 and looks it got struck ☹
GET cluster/health?pretty
{
"cluster\_name" : "OConnectElasticSearch",
"status" : "green",
"timed\_out" : false,
"number\_of\_nodes" : 9,
"number\_of\_data\_nodes" : 6,
"active\_primary\_shards" : 20472,
"active\_shards" : 40944,
**"relocating\_shards" : 6,**
"initializing\_shards" : 0,
"unassigned\_shards" : 0,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 526,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 549051,
"active\_shards\_percent\_as\_number" : 100.0
}
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [October 19, 2020, 1:50pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/13 "2020-10-19T13:50:25Z")
> [@muthug](#):
>
> True!! I am working on that. Soon it will be addressed and moved to new ELK stack 7.9.
You don't need to wait for an upgrade to address that problem.
From tomorrow, just start new indices with only one shard and one replica.
Also you have very old indices in your cluster. Like this one `vdc_b_e063-2019.06.01` which has more than one year.
Do you still need those indices?
---
### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 1:54pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/14 "2020-10-19T13:54:57Z")
I haves deleted those indices. but now relocating shards are got struck.
```auto
vdc_b_e061-2020.05.05 4 r RELOCATING 0 159b 0.46.xx.xx OConnectDataNode06 -> 10.46.xx.xx jmmTwCkxQ_mQbLLllcd_Cg OConnectDataNode05
vdc_b_e061-2020.05.05 3 r RELOCATING 0 160b 0.46.xx.xx OConnectDataNode04 -> 10.46.xx.xx jmmTwCkxQ_mQbLLllcd_Cg OConnectDataNode05
vdc_b_e061-2020.05.05 5 p RELOCATING 0 159b 0.46.xx.xx OConnectDataNode04 -> 10.46.xx.xx jmmTwCkxQ_mQbLLllcd_Cg OConnectDataNode05
vdc_b_e061-2020.05.05 1 r RELOCATING 0 160b 0.46.xx.xx OConnectDataNode04 -> 10.46.xx.xx jmmTwCkxQ_mQbLLllcd_Cg OConnectDataNode05
vdc_b_e061-2020.05.05 2 p RELOCATING 1 13.2kb 0.46.xx.xx OConnectDataNode03 -> 10.46.xx.xx jmmTwCkxQ_mQbLLllcd_Cg OConnectDataNode05
vdc_b_e061-2020.05.05 0 r RELOCATING 0 160b 0.46.xx.xx OConnectDataNode04 -> 10.46.xx.xx jmmTwCkxQ_mQbLLllcd_Cg OConnectDataNode05
```
---
### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 1:57pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/15 "2020-10-19T13:57:26Z")
If i fix this relocating shard issue then i am good. Bcoz past 5 days i am working this issue today finally its become green but with relocating shads 6
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [October 19, 2020, 2:07pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/16 "2020-10-19T14:07:41Z")
> [@muthug](#):
>
> I haves deleted those indices.
What is the **full** output of:
```auto
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v
```
---
### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 2:30pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/17 "2020-10-19T14:30:17Z")
> [@dadoonet](#):
>
> `/_cat/health?v`
GET /\_cat/nodes?v
host ip heap.percent ram.percent load node.role master name
10.46.xx.xx 10.46.xx.xx 14 58 0.27 d m OConnectDataNode05
10.46.xx.xx 10.46.xx.xx 30 59 0.54 d m OConnectDataNode01
10.46.xx.xx 10.46.xx.xx 48 55 0.96 - \* OConnectManagementNode
10.46.xx.xx 10.46.xx.xx 13 57 0.43 d m OConnectDataNode02
10.46.xx.xx 10.46.xx.xx 15 57 0.44 d m OConnectDataNode03
10.46.xx.xx 10.46.xx.xx 61 57 0.20 - - OConnectClientNode02
10.46.xx.xx 10.46.xx.xx 23 57 0.63 d - OConnectDataNode04
10.46.xx.xx 10.46.xx.xx 53 0 -1.00 - - OConnectClientNode01
10.46.xx.xx 10.46.xx.xx 59 57 0.54 d m OConnectDataNode06
health?v
{
"cluster\_name" : "BPOConnectElasticSearch",
"status" : "green",
"timed\_out" : false,
"number\_of\_nodes" : 9,
"number\_of\_data\_nodes" : 6,
"active\_primary\_shards" : 20472,
"active\_shards" : 40944,
"relocating\_shards" : 6,
"initializing\_shards" : 0,
"unassigned\_shards" : 0,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 2409,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 2561610,
"active\_shards\_percent\_as\_number" : 100.0
}
GET /\_cat/indices?v
>
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [October 19, 2020, 2:44pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/18 "2020-10-19T14:44:40Z")
I can still see a lot of data from 2019.
> [@muthug](#):
>
> I haves deleted those indices.
So I don't think you did.
---
### Author:  [@muthug](https://discuss.elastic.co/u/muthug)
#### Post date: [October 19, 2020, 2:47pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/19 "2020-10-19T14:47:12Z")
As per client agreement we should maintain 2 years log. What are the shards are shown unassigned i have deleted those, not all 2019.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [October 19, 2020, 3:09pm UTC](https://discuss.elastic.co/t/initializing-shards-got-struck/252511/20 "2020-10-19T15:09:16Z")
How can I know that you did not follow the advices although you said you did?
Anyway, I deeply agree with @Christian_Dahlqvist's advices:
> [@Christian\_Dahlqvist](#):
>
> I would recommend you start addressing this ASAP (go to a single primary shard in your index templates, consolidate indices, switch from daily to e.g. monthly indices where the size is small) as it is just otherwise going to complicate the migration.
[Next page](https://discuss.elastic.co/t/initializing-shards-got-struck/252511.md?page=2)