# Still getting 'max open shards' error after re-indexing and decreasing shards to 850 **URL:** \ **Category:** Elasticsearch\ **Created:** [July 15, 2020, 10:40pm UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369 "2020-07-15T22:40:02Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![anda](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@anda](https://discuss.elastic.co/u/anda)\ **Post date:** [July 15, 2020, 10:40pm UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/1 "2020-07-15T22:40:02Z")
Hello, I'm running ELK 7.4 and have been getting this error in Logstash logs for basically every event that's being processed: ```auto [2020-07-15T15:24:06,789][WARN][logstash.outputs.elasticsearch][enm] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ssoinstr-2020.29", :_type=>"_doc", :routing=>nil}, #], :response=>{"index"=>{"_index"=>"ssoinstr-2020.29", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}} ``` Before, I had a total of ~5000 shards for my 7 node cluster (1 replica per index). Despite the error in the Logstash logs, I could still see the data being indexed via Kibana. But I wanted to get rid of the recurring log errors anyways so I followed some tips online to lower the number of shards After re-indexing my data, the number of shards for my cluster decreased to ~850. I confirmed the number of shards using: `GET _cluster/stats?filter_path=indices.shards.total` I check the logs and was still getting the error. I restarted the Logstash instances but continued to get the same errors (errors were still saying I had 999 open shards). As a last ditch effort, I increased the number of max open shards to 2000 using the API, but continued to get the error. Does anyone have any experience with this? While the error doesn't actually seem to be affecting performance of the cluster, it's very annoying and clutters the log when I'm trying to troubleshoot. Thank you!! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 16, 2020, 12:18am UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/2 "2020-07-16T00:18:48Z")
Is there similar logs in Elasticsearch? ---
**Author:** ![anda](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@anda](https://discuss.elastic.co/u/anda)\ **Post date:** [July 16, 2020, 6:19pm UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/3 "2020-07-16T18:19:03Z")
No - I'm not getting any errors in the Elasticsearch logs ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 16, 2020, 9:24pm UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/4 "2020-07-16T21:24:09Z")
How did you lower the number of shards? Deletion? Reindexing? Merge indices? ---
**Author:** ![anda](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@anda](https://discuss.elastic.co/u/anda)\ **Post date:** [July 16, 2020, 10:24pm UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/5 "2020-07-16T22:24:37Z")
I re-indexed a large number of smaller indices into a fewer number of larger indices, and then deleted the original smaller indices ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [July 17, 2020, 3:11am UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/6 "2020-07-17T03:11:26Z")
There is also per node shard limit; default is 1K, which looks like the error you get 999/1000. Via cluster.max\_shards\_per\_node - set that higher to see if your error goes away and look at shards per node which is hard to get - I think really only by getting a shard list GET /\_cat/shards and counting by node name or ID. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 17, 2020, 3:30am UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/7 "2020-07-17T03:30:56Z")
Do you have any closed indices? ---
**Author:** ![anda](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@anda](https://discuss.elastic.co/u/anda)\ **Post date:** [July 17, 2020, 4:27am UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/8 "2020-07-17T04:27:36Z")
On my monitoring page it says I have 822 open shards total. To double check, I saved the output to GET /\_cat/shards and wrote a script to count the total number of shards and how many were on each node. I got 822 total and 274 on each node since I have 3 data nodes. So it seems like for whatever reason Logstash hasn't identified the change in number of open shards per node. It continues to tell me that it can't index the event because there are too many open shards, but the event gets indexed anyways. Color me confused. ---
**Author:** ![anda](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@anda](https://discuss.elastic.co/u/anda)\ **Post date:** [July 17, 2020, 4:27am UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/9 "2020-07-17T04:27:56Z")
No closed indices ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [July 17, 2020, 5:42am UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/10 "2020-07-17T05:42:49Z")
Logstash doesn't care; the error is a rejection from Elasticsearch - with 822 then you are no exceeding 1K of course, so seems odd. To 'add' shards on ssoinstr-2020.29 it must be creating that index I assume so this does seem strange; I guess I suggest raising the cluster.max\_shards\_per\_node anyway as that defaults to 1K which is the number cited in your error to see if it goes away - your cluster is green, right? Also very odd the data is indexed anyway; I guess it might try to create the shard on another node, but the error mentions the cluster which is also odd. You should confirm your cluster max - what "max open shards" setting are you setting? The only setting I know is per node but this makes no sense if your cluster limit it 1K on 7 nodes; someone would have had to set this to 1K/7; very low. So verify max\_shards\_per\_node hasn't been changed by someone to something lower. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 14, 2020, 5:50am UTC](https://discuss.elastic.co/t/still-getting-max-open-shards-error-after-re-indexing-and-decreasing-shards-to-850/241369/11 "2020-08-14T05:50:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.