# Shards management in ElasticSearch

**URL:** <https://discuss.elastic.co/t/shards-management-in-elasticsearch/291252>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [December 8, 2021, 6:53pm UTC](https://discuss.elastic.co/t/shards-management-in-elasticsearch/291252 "2021-12-08T18:53:13Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Akanksha\_Pandey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akanksha_pandey/32/89539_2.png) [@Akanksha\_Pandey](https://discuss.elastic.co/u/Akanksha_Pandey)\
**Post date:** [December 8, 2021, 6:53pm UTC](https://discuss.elastic.co/t/shards-management-in-elasticsearch/291252/1 "2021-12-08T18:53:13Z")

</div>

I'm using Elasticsearch version 7.9.2 as a single node docker cluster for my production as well as development environment. Every once in a while I'm getting these two errors frequently.

**Error1:**

```auto
ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: validation_exception [reason]: 'Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2000]/[2000] maximum shards open

```

and **Error2:** `unassigned shards`:

```auto
{
  "cluster_name" : "docker-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2227,
  "active_shards" : 2227,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 14,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 99.37527889335118
}

```

So I performed these settings in the Kibana DevTools console.

```auto

PUT /_cluster/settings
{"transient":{"cluster.max_shards_per_node":3000}}

PUT /*/_settings
{
"index" : {
"number_of_replicas":0,
"auto_expand_replicas": false
}
}

PUT /_cluster/settings
{

    "transient" : {

        "cluster.routing.allocation.enable" : "all"

    }

}

PUT /_template/default_template
{

  "index_patterns": ["*"],

  "settings": {

    "index": {

      "number_of_replicas": 0

    }

  }

}

```

It helped me to resolve the issues at that moment only. After a while these issues come back again. Can anyone help me how to integrate these settings in `elasticsearch.yml` file and automate the settings so they do not come again as this is hampering the production systems.

Please anyone help me!

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 8, 2021, 7:30pm UTC](https://discuss.elastic.co/t/shards-management-in-elasticsearch/291252/2 "2021-12-08T19:30:31Z")

</div>

[Having lots of small shards is very, very inefficient](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) and can cause both performance and stability issues. I woiuld therefore strongly recommend that you change how you shard your data rather than increase this limit and reduce the shard count dramatically. A good average shard size to aim for is in the range of a few GB up to a few tens of GB.

---

<div class="post-metadata">

**Author:** ![Akanksha\_Pandey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akanksha_pandey/32/89539_2.png) [@Akanksha\_Pandey](https://discuss.elastic.co/u/Akanksha_Pandey)\
**Post date:** [December 8, 2021, 8:02pm UTC](https://discuss.elastic.co/t/shards-management-in-elasticsearch/291252/3 "2021-12-08T20:02:49Z")

</div>

@Christian_Dahlqvist - thanks for the response. Can you help me how to change the shard size in `elasticsearch.yml` file? And what should be that optimal shard size?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 8, 2021, 8:51pm UTC](https://discuss.elastic.co/t/shards-management-in-elasticsearch/291252/4 "2021-12-08T20:51:54Z")

</div>

It is not set in config files. It is driven by how you are creating index names when you are indexing.

---

<div class="post-metadata">

**Author:** ![Akanksha\_Pandey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akanksha_pandey/32/89539_2.png) [@Akanksha\_Pandey](https://discuss.elastic.co/u/Akanksha_Pandey)\
**Post date:** [December 9, 2021, 2:10am UTC](https://discuss.elastic.co/t/shards-management-in-elasticsearch/291252/5 "2021-12-09T02:10:41Z")

</div>

Thanks for responding.

I'm creating index names in `filebeat.yml` file. Should I need to put shards configurations here? and if yes, please suggest!

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 9, 2021, 4:49am UTC](https://discuss.elastic.co/t/shards-management-in-elasticsearch/291252/6 "2021-12-09T04:49:41Z")

</div>

You need to change how you name your indices so you create fewer.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 6, 2022, 4:49am UTC](https://discuss.elastic.co/t/shards-management-in-elasticsearch/291252/7 "2022-01-06T04:49:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.