# ELK cluster is not stable due to several issues related to shards and JVM configuration **URL:** \ **Category:** Elasticsearch\ **Created:** [August 11, 2021, 12:30pm UTC](https://discuss.elastic.co/t/elk-cluster-is-not-stable-due-to-several-issues-related-to-shards-and-jvm-configuration/281059 "2021-08-11T12:30:10Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Dawood](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Dawood](https://discuss.elastic.co/u/Dawood)\ **Post date:** [August 11, 2021, 12:30pm UTC](https://discuss.elastic.co/t/elk-cluster-is-not-stable-due-to-several-issues-related-to-shards-and-jvm-configuration/281059/1 "2021-08-11T12:30:10Z")
Hi, First it is not the first time i am sharing the issues that i am encountering and hopefully this time somebody will help me with that !!! So i have an ELK cluster that is running on premise on a rancher k8s based on 3 VMs with the following resources: 32 RAM and 8 CPU and unlimited storage ( volume mount). The cluster is created to save automation python logs that are directed to Logstash APP already structured using logstash\_async module. Every python script creating new unique index and its logs will create new indices with same index created which could be hundred of logs. I deployed logstash ( 3 replicas),elasticsearch(3 replicas) and kibana(1 replica) using Helm Charts ( I can share them if needed). I did not configure anything in any chart something that related to shards. The application working fine for a period of time let's say one month and then everything starts to break mainly in the elasticsearch pods: ```auto total shards, but this cluster currently has [3000]/[3000] maximum shards open;"}}}} > 7/28/2021 6:09:42 PM [2021-07-28T15:09:42,731][WARN][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"qascreencapture-1627484927937893928", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"qascreencapture-1627484927937893928", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [3000]/[3000] maximum shards open;"}}}} ``` I fixed that by the following :curl -XPUT -H 'Content-Type: application/json' 'IP-OF-ELASTIC-SERVER:9200/\_cluster/settings' -d '{ "persistent" : {"cluster.max\_shards\_per\_node" : 5000}}' I am pretty sure that this is not the way to correct to do that and there some other solutions that i need here your suggestions. Then after that everything worked well till i got new errors related to JVM and HEAP but mainly related to the change i made with shards. Please help me. 10x, Dawoods ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 11, 2021, 12:41pm UTC](https://discuss.elastic.co/t/elk-cluster-is-not-stable-due-to-several-issues-related-to-shards-and-jvm-configuration/281059/2 "2021-08-11T12:41:13Z")
> [@Dawood](#): > > Every python script creating new unique index and its logs will create new indices with same index created which could be hundred of logs. This seems to be a very inefficient and wasteful way to index data into Elasticsearch and will not scale well each shard has overhead and increases the size of the cluster state. Please read [this blog post on sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) and find a way to not create a new index per script for low volumes of data. If you want to store data for a long time or expect large volumes you need to make sure that your shard sizes are at least a few tens of GB in size. ---
**Author:** ![Dawood](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Dawood](https://discuss.elastic.co/u/Dawood)\ **Post date:** [August 12, 2021, 6:25am UTC](https://discuss.elastic.co/t/elk-cluster-is-not-stable-due-to-several-issues-related-to-shards-and-jvm-configuration/281059/3 "2021-08-12T06:25:42Z")
Thanks 🙂 I will change the way i am indexing my logs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 9, 2021, 6:26am UTC](https://discuss.elastic.co/t/elk-cluster-is-not-stable-due-to-several-issues-related-to-shards-and-jvm-configuration/281059/4 "2021-09-09T06:26:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.