# Shard limit hit
**URL:** https://discuss.elastic.co/t/shard-limit-hit/123606
**Category:** Elasticsearch
**Created:** [March 12, 2018, 5:45pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606 "2018-03-12T17:45:36Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@jmgarcia](https://discuss.elastic.co/u/jmgarcia)
#### Post date: [March 12, 2018, 5:45pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/1 "2018-03-12T17:45:36Z")
Well to get down to it we migraded servers this weekend. My script worked before but now it doesnt. I keep on getting an error that
> "[{'error': {'root\_cause': [{'type': 'illegal\_argument\_exception', 'reason': 'Trying to query 1536 shards, which is over the limit of 1000. This limit exists because querying many shards at the same time can make the job of the coordinating node very CPU and/or memory intensive. It is usually a better idea to have a smaller number of larger shards. Update [action.search.shard\_count.limit] to a greater value if you really want to query that many shards at the same time.'}]"
I have been trying all day to get either action.search.shard\_count.limit or max\_concurrent\_shard\_requests working and cant quite figure it out. Here is a snip of my code...
> {
> "size": 0,
> "query": {
> "query\_string": {
> "query": fleCont
> }
> },
> "aggs": {
> "per\_scott": {
> "terms": {
> "script": {
> "lang": "painless",
> "inline": "doc['src\_ip'].value + ',' + doc['dst\_ip'].value + ',' + doc['dst\_port'].value + ',' + doc['proto'].value + ',' + doc['devicename'].value + ',' + doc['policy\_id'].value"
> },
> "size": 10000,
> }
> }
> }
> }
---
### Author:  [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)
#### Post date: [March 12, 2018, 5:56pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/2 "2018-03-12T17:56:21Z")
Funny, I just hit this limit last week. It turned out I was querying way more indices than I needed. My query used a wildcard `myindex-*`. With this, I was querying months of data when I only needed the last thirty days. I changed my logic to account for the date. So now I'm searching `myindex-2018.03.*,myindex-2018.02.*`, and lightening the load on my es stack. This may not apply to your case, but I thought I'd share my anecdote just in case.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [March 12, 2018, 6:00pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/3 "2018-03-12T18:00:25Z")
That's definitely a good solution.
---
### Author:  [@jmgarcia](https://discuss.elastic.co/u/jmgarcia)
#### Post date: [March 12, 2018, 6:00pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/4 "2018-03-12T18:00:46Z")
Hey thanks pixel but yeah, I need to search all of logstash. It is just strange why my admin would have reduced the allowed shard size
---
### Author:  [@jmgarcia](https://discuss.elastic.co/u/jmgarcia)
#### Post date: [March 12, 2018, 6:02pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/5 "2018-03-12T18:02:36Z")
I agree it is but we have 200+ logstash logs to dig through. I think it would be more detrimental to the server to feed each logstash in, connect, search, disconnect etc than just upping the shard counts.
---
### Author:  [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)
#### Post date: [March 12, 2018, 6:22pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/6 "2018-03-12T18:22:58Z")
1000 is the default. Your admin either neglected to migrate this custom setting over, or more likely, your admin may have increased the number of shards per index on your new cluster.
Ideally, you will want to keep within the recommended settings and rewrite logic around the limitations. That said, I think I found a post with the answer you are looking for. Changing the limit is a cluster setting that will need to be configured by an administrator:
> [@Action.search.shard\_count.limit exceeded](https://discuss.elastic.co/t/action-search-shard-count-limit-exceeded/62891):
>
> Can someone please help locate the file containing this parameter action.search.shard\_count.limit I trying to parse a single log file of around 3.2 GB using ELK stack. The same has got indexed and when I am trying to discover indexed logs in Kibana I get the following error. Error: Discover: Trying to query 2051 shards, which is over the limit of 1000. This limit exists because querying many shards at the same time can make the job of the coordinating node very CPU and/or memory intensive. It…
---
### Author:  [@jmgarcia](https://discuss.elastic.co/u/jmgarcia)
#### Post date: [March 12, 2018, 7:19pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/7 "2018-03-12T19:19:37Z")
I am looking into how to rewrite my logic but I am lost. Do you have any suggestions?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [March 12, 2018, 7:29pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/8 "2018-03-12T19:29:36Z")
First thing is why using so many shards?
May I suggest you look at the following resources about sizing:
[https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)
> **[How many shards should I have in my Elasticsearch cluster?
| Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**
>
> Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet...
> **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)**
>
> Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type « Black Friday » (Vendredi noir) ou période de Noël peuvent pousser...
---
### Author:  [@jmgarcia](https://discuss.elastic.co/u/jmgarcia)
#### Post date: [March 12, 2018, 7:39pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/9 "2018-03-12T19:39:18Z")
I do not manage this server. I was told to "search Elasticsearch for X and get the results". So that is what I am doing. I dont mean to sound mean or apathetic but 1) This just started happening today (when Friday it wasnt -\> yes we migrated over the weekend) and 2) I am at a disconnect ad being an admin on a elasticsearch database because I was literally thrown into this projected after I said I dont know it.
I will email the admin and see what they say but I feel like I need a little bit better ground to stand on first. I will watch the video and read some more but still, I am not sure I will have enough.
---
### Author:  [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)
#### Post date: [March 12, 2018, 7:48pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/10 "2018-03-12T19:48:03Z")
@dadoonet unfortunately I don't think OP is the admin, so tweaking his cluster may not be an option for him.
@jmgarcia how many indices are you hitting when you make your query? Are your indices historical as-in daily or monthly (i.e. myindex-2018.03)? Are you using wildcards in your index request (i.e. myindex-\*)? If so, the workaround for this case would be something like this:
Establish the date range you need to query (i.e. last 24 months)
Divide this range into months (2018.03, 2018.02, 2018.01, 2017.12, etc)
Then join these months into acceptable chunks that fly under your limit:
query1: `https://myeshost:9200/myindex-2018.01*,myindex-2017.12*`
query2: `https://myeshost:9200/myindex-2018.03*,myindex-2018.02*`
....etc....
Finally, you would populate a master dict where you manually aggregate the multiple queries into a single search result.
---
### Author:  [@jmgarcia](https://discuss.elastic.co/u/jmgarcia)
#### Post date: [March 12, 2018, 7:52pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/11 "2018-03-12T19:52:17Z")
You are correct @pixelrebel I am not an admin.
When I do a the cat command I get a total of 128 different logstash that I need to search through. They are moved to a different server every ~30 days and there is no way I will be able to access those servers.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 12, 2018, 8:57pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/12 "2018-03-12T20:57:20Z")
This limit on the number of shards that can be queried [was introduced in Elasticsearch 5.x](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/breaking_50_search_changes.html#_search_shard_limit), which might have been what you might have migrated to over the weekend.
As David stated, it does look like you have a lot of shards. If you can provide the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cluster-stats.html) we can get a better idea about the state of the cluster.
---
### Author:  [@jmgarcia](https://discuss.elastic.co/u/jmgarcia)
#### Post date: [March 13, 2018, 5:17pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/13 "2018-03-13T17:17:09Z")
Alright, after more digging and talking with the admin, they changed the requirements so now I am not hitting the shard limit. A couple are close (I think one returns 989) but for now the issue is solved.
Right now I need to scrub the cluser stats too much @Christian_Dahlqvist and I dont have enough time to do that and post the results in here. Tomorrow or the following day I will though. In the future I know I will be running into this problem again so all I need is a little more time to sort other stuff out then we will be golden.
Thank you to everybody who helped.
Adios.... for now.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 13, 2018, 5:19pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/14 "2018-03-13T17:19:18Z")
The output of the cluster stats (not state) API typically does not contain anything particularly sensitive. What is it that needs to be scrubbed?
---
### Author:  [@jmgarcia](https://discuss.elastic.co/u/jmgarcia)
#### Post date: [March 14, 2018, 1:40pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/15 "2018-03-14T13:40:27Z")
They dont want the server names or anything like it released, including the logstash specific names. I will do my best to get the results today.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 11, 2018, 1:40pm UTC](https://discuss.elastic.co/t/shard-limit-hit/123606/16 "2018-04-11T13:40:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.