Shards management in ElasticSearch

[Akanksha_Pandey]

I'm using Elasticsearch version 7.9.2 as a single node docker cluster for my production as well as development environment. Every once in a while I'm getting these two errors frequently.

Error1:

ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: validation_exception [reason]: 'Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2000]/[2000] maximum shards open

and Error2: unassigned shards:

{
  "cluster_name" : "docker-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 2227,
  "active_shards" : 2227,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 14,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 99.37527889335118
}

So I performed these settings in the Kibana DevTools console.

PUT /_cluster/settings
{"transient":{"cluster.max_shards_per_node":3000}}


PUT /*/_settings
{
"index" : {
"number_of_replicas":0,
"auto_expand_replicas": false
}
}


PUT /_cluster/settings
{

    "transient" : {

        "cluster.routing.allocation.enable" : "all"

    }

}


PUT /_template/default_template
{

  "index_patterns": ["*"],

  "settings": {

    "index": {

      "number_of_replicas": 0

    }

  }

}

It helped me to resolve the issues at that moment only. After a while these issues come back again. Can anyone help me how to integrate these settings in elasticsearch.yml file and automate the settings so they do not come again as this is hampering the production systems.

Please anyone help me!

Thanks in advance.

[Christian_Dahlqvist]

Having lots of small shards is very, very inefficient and can cause both performance and stability issues. I woiuld therefore strongly recommend that you change how you shard your data rather than increase this limit and reduce the shard count dramatically. A good average shard size to aim for is in the range of a few GB up to a few tens of GB.

[Akanksha_Pandey]

@Christian_Dahlqvist - thanks for the response. Can you help me how to change the shard size in elasticsearch.yml file? And what should be that optimal shard size?

[Christian_Dahlqvist]

It is not set in config files. It is driven by how you are creating index names when you are indexing.

[Akanksha_Pandey]

Thanks for responding.

I'm creating index names in filebeat.yml file. Should I need to put shards configurations here? and if yes, please suggest!

[Christian_Dahlqvist]

You need to change how you name your indices so you create fewer.