# Shards getting unassigned Randomly
**URL:** https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699
**Category:** Elasticsearch
**Tags:** docker
**Created:** [February 3, 2020, 9:33pm UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699 "2020-02-03T21:33:48Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@rokarajan](https://discuss.elastic.co/u/rokarajan)
#### Post date: [February 3, 2020, 9:33pm UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699/1 "2020-02-03T21:33:48Z")
Hello ELK forum members,
I'm new on this forum and would like to share a problem we are currently facing with ELK.
We have been running ELK in our Kubernetes(EKS) for few months with version 7.2.0. We have 12 nodes in our cluster( 3xElastic, 6xData, 3xClients) plus 5xKibana. Main purpose of the cluster is to centralise our logs from different platforms. For example; kubernetes logs are pushed using fluentbit agent and legacy applications logs are pushed to the cluster using lambda functions.
We collect in average 200GB data per day and the data retention varies depending on the index 4 days to 30 days. We have about 1200 indices and 4000 shards with about 5TB data. Each nodes(Data and Elastic) have 16GB memory with 8GB assigned to Java.
Now the problem:
We have been having issue with shards getting unassigned randomly without any particular pattern. Make sense; during the index rollout time(daily) there are many new indexes created and takes a some time for the cluster to assign shards to the nodes but it is happening outside that window. We have also isolated this is not related to the curator. My question here in the forum; has anyone experienced similar situation? Is that related to our design? - about 650shards/data node? Any help is really appreciated.
Thanks,
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [February 4, 2020, 12:32am UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699/2 "2020-02-04T00:32:12Z")
Welcome!
You probably have too many shards per node.
We normally expect 20 shards max per gb of heap. With 8gb, that would mean no more than 160 shards per node.
Also at this size, I'd probably recommend having dedicated master eligible nodes.
May I suggest you look at the following resources about sizing:
[https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)
> **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**
>
> If you are looking for practical guidelines around how many indices and shards to have in your cluster, this blog post will help you avoid common pitfalls.
https://www.slideshare.net/slideshow/embed_code/key/vR0XKDq4TGa77z
And [https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right](https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right)
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 4, 2020, 1:21am UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699/3 "2020-02-04T01:21:20Z")
> [@rokarajan](#):
>
> We have been having issue with shards getting unassigned randomly without any particular pattern.
The server logs likely contain a lot of useful information about why this is happening, but without those logs we can only really speculate. Can you share the logs from all nodes from around the time of the incident? Please redact as little as possible, but if you must redact something then please make it clear it has been redacted. Use [https://gist.github.com/](https://gist.github.com/) since there will be too much information to quote in full here.
---
### Author:  [@rokarajan](https://discuss.elastic.co/u/rokarajan)
#### Post date: [February 4, 2020, 2:29am UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699/4 "2020-02-04T02:29:34Z")
Thanks @dadoonet I will go through the video and review our architecture. Thanks for sharing that.
---
### Author:  [@rokarajan](https://discuss.elastic.co/u/rokarajan)
#### Post date: [February 4, 2020, 2:50am UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699/5 "2020-02-04T02:50:00Z")
Hi @DavidTurner thanks for your reply. Is there any particular logs I need to filter for this one? Master logs or the Data Nodes logs? I could see heaps of logs generated but unsure which one would be helpful?
When it happened recently i managed to grab the output of the \_cluster/allocation/explain output as below.
>
thanks
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 4, 2020, 8:18am UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699/6 "2020-02-04T08:18:37Z")
As I said, logs from _all nodes_. Don't worry about their size, we can help you find the right information amongst the noise.
---
### Author:  [@rokarajan](https://discuss.elastic.co/u/rokarajan)
#### Post date: [February 5, 2020, 12:19am UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699/7 "2020-02-05T00:19:56Z")
> [@DavidTurner](#):
>
> As I said, logs from _all nodes_. Don't worry about their size, we can help you find the right information amongst the noise.
Hi @DavidTurner, I have uploaded the logs here.
> **[Google Drive: Sign-in](https://accounts.google.com/v3/signin/identifier?continue=https%3A%2F%2Fdrive.google.com%2Fdrive%2Ffolders%2F1EAyg_AR8XsTvDSyKReq1K0UA1rQZ7VaO&followup=https%3A%2F%2Fdrive.google.com%2Fdrive%2Ffolders%2F1EAyg_AR8XsTvDSyKReq1K0UA1rQZ7VaO&ifkv=ASKXGp2g-C0TYxbeYyqP5BxfC1Y6R0VrWm6jmY3KxytEXCwmwTksASPDEdhNjcJhbha-pIXCjtBk8w&osid=1&passive=1209600&service=wise&flowName=GlifWebSignIn&flowEntry=ServiceLogin&dsh=S789278931%3A1701352963488282&theme=glif)**
>
> Access Google Drive with a Google account (for personal use) or Google Workspace account (for business use).
Looking forward to hear back from you.
Thanks
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 5, 2020, 9:58am UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699/8 "2020-02-05T09:58:27Z")
The logs from the elected master only contain about three minutes of data, and those three minutes don't include any nodes joining or leaving the cluster:
```auto
$ cat elastic-apse2-master-2_logging_elasticsearch-0306f7bbc0cdf35ab47100db1569bf1412a5feb1c446132826d0cdff7bc879b6.log | sed -n '1p;$p' | jq .time
"2020-02-04T23:02:07.397589542Z"
"2020-02-04T23:04:52.986172572Z"
```
It looks like you have a Kibana instance pointed at the master nodes too. I recommend not doing that - it's generating a lot of noise in the logs, and the master nodes should not be used for external requests.
You might also like to disable monitoring for now, since that is also generating a lot of noise in the logs.
---
### Author:  [@rokarajan](https://discuss.elastic.co/u/rokarajan)
#### Post date: [February 5, 2020, 10:46pm UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699/9 "2020-02-05T22:46:18Z")
Thanks @DavidTurner!
It's really tricky to get the logs from all nodes when the problem occurs; I will probably try logging them to ELK itself so that I get some persistent logs.
Yes currently; we are pointing Kibana to the master node and we not making the good use of client nodes; do you think its good idea pointing Kibana to them?
In regards to the monitoring, are you referring to the Kibana xpack monitoring? We have below setup for them
```
xpack.monitoring.enabled: true
xpack.monitoring.kibana.collection.enabled: true
xpack.monitoring.ui.enabled: true
xpack.monitoring.ui.container.elasticsearch.enabled: true
```
Thanks,
Raj
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 4, 2020, 10:46pm UTC](https://discuss.elastic.co/t/shards-getting-unassigned-randomly/217699/10 "2020-03-04T22:46:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.