# Search phase execution exception with reason all shards failed

**URL:** <https://discuss.elastic.co/t/search-phase-execution-exception-with-reason-all-shards-failed/216797>\
**Category:** Kibana\
**Created:** [January 28, 2020, 9:42am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-with-reason-all-shards-failed/216797 "2020-01-28T09:42:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Voula\_Mikr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voula_mikr/32/58451_2.png) [@Voula\_Mikr](https://discuss.elastic.co/u/Voula_Mikr)\
**Post date:** [January 28, 2020, 9:42am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-with-reason-all-shards-failed/216797/1 "2020-01-28T09:42:58Z")

</div>

Hi!

I am trying to add a sub-bucket in an kibana histogram and i get the bellow:

"type":"search\_phase\_execution\_exception","reason":"all shards failed","phase":"query","grouped":true,"failed\_shards":[

{"shard":0,"index":"data-20200123",  
"node":"CXFldNupTpObai1jt88wDg","reason":{"type":"too\_many\_buckets\_exception","reason":"Trying to create too many buckets. Must be less than or equal to:  
[10000] but was [10001]. This limit can be set by changing the [search.max\_buckets] cluster level setting.","max\_buckets":10000}},

{"shard":0,"index":"data-20200127",  
"node":"CXFldNupTpObai1jt88wDg","reason":{"type":"too\_many\_buckets\_exception","reason":"Trying to create too many buckets.  
Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max\_buckets] cluster level setting.","max\_buckets":10000}},

{"shard":0,"index":"data-a-20200123","node":"CXFldNupTpObai1jt88wDg","reason":{"type":"too\_many\_buckets\_exception","reason":"Trying to create too many buckets.  
Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max\_buckets] cluster level setting.","max\_buckets":10000}},

{"shard":0,"index":"data-20200127","node":"rF8XLfPSSkSUBUdIgr9Lzw","reason":{"type":"too\_many\_buckets\_exception","reason":"Trying to create too many buckets.  
Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max\_buckets] cluster level setting.","max\_buckets":10000}}  
]},

"status":503}

"search\_phase\_execution\_exception","reason":"all shards failed"

I guess that during the sub-bucket aggregation I get more than 10000 buckets from elasticsearch as answer.  
How can I get over this problem?  
Could be a solution to the proble to increase bucket size, or to increase number of buckets or to reduce size of shards. Is that feasible for elasticsearch or kibana?

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\
**Post date:** [January 28, 2020, 12:54pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-with-reason-all-shards-failed/216797/2 "2020-01-28T12:54:03Z")

</div>

It can be done, not really recommended as if you don't have enough resources for the ES cluster, there could be problems.  
This limit can be set by changing the [search.max\_buckets] cluster level setting.","max\_buckets":10000

This is the API to change the settings: [https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html)  
You can also do it in the elasticsearch.yml file.

---

<div class="post-metadata">

**Author:** ![Voula\_Mikr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voula_mikr/32/58451_2.png) [@Voula\_Mikr](https://discuss.elastic.co/u/Voula_Mikr)\
**Post date:** [January 28, 2020, 3:25pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-with-reason-all-shards-failed/216797/3 "2020-01-28T15:25:15Z")

</div>

Thank you Marius for your answer.

Since I have only 4 indices with total 200 docs and the distinguished values for the sub-bucket aggregation are 30 how it exceeds the max\_bucket (10000)?

Does the number of buckets has to do with all the sub-bucket aggregations cumulatively?

---

<div class="post-metadata">

**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\
**Post date:** [January 28, 2020, 3:30pm UTC](https://discuss.elastic.co/t/search-phase-execution-exception-with-reason-all-shards-failed/216797/4 "2020-01-28T15:30:47Z")

</div>

It depends on the number of buckets in the timestamp as well. It woud be 30x nr of time buckets. And if you used a small interval for the bucket, across a large time period, it can easily go over 10000.

---

<div class="post-metadata">

**Author:** ![Voula\_Mikr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voula_mikr/32/58451_2.png) [@Voula\_Mikr](https://discuss.elastic.co/u/Voula_Mikr)\
**Post date:** [January 29, 2020, 7:08am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-with-reason-all-shards-failed/216797/5 "2020-01-29T07:08:31Z")

</div>

Thank you Marius!

Is there any "formula" to calculate the buckets knowing the number of distinguished values of the sub-bucket aggregations (3 sub-buckets (terms) in a kibana histogram), the interval and the time period?  
For example:  
subbucket A : 30 distinguished values  
subbucket B: 5 distinguished values  
subbucket C: 10 distinguished values  
and all this requested 80 times in a time period.  
A multiplication of all these? (I guess that this way I will get the max possible but not the real number)

Or I have to count all the combinations from these 3 subbuckets I get for each moment in the time period and add all these numbers?  
For example:  
9:00 : results=8 (1 doc with aa-bb-cc / 4 docs with aa -bb -cb / 3 docs with ab -bb -cc)  
9:15 : results=9 (1 doc with aa-bb-cc / 4 docs with aa -bb -cb / 2 docs with ab -bb - cc / 2 docs with ab-bb-ca)  
9:30 : results=3 (1 doc with ad-bb-ca / 1 doc with aa -bb -cb / 1 docs with ab -bf -ca)

(Where e.g1 doc with ad-bb-ca means that the query for subbucket A has as result the value ad  
the query for subbucket B has as result the value bb  
and the query for subbucket C has as result the value ca  
)

So in this way the buckets for time period [9:00-9:30] will be 8+9+3 ?  
😕

Thank you again!!

---

<div class="post-metadata">

**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\
**Post date:** [January 29, 2020, 11:11am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-with-reason-all-shards-failed/216797/6 "2020-01-29T11:11:44Z")

</div>

I really don't know how elasticsearch calculates it, but I do guess it's the first one with the max possible as it doesn't know hot many combinations there are until it starts calculating them.

---

<div class="post-metadata">

**Author:** ![Voula\_Mikr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voula_mikr/32/58451_2.png) [@Voula\_Mikr](https://discuss.elastic.co/u/Voula_Mikr)\
**Post date:** [January 29, 2020, 11:13am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-with-reason-all-shards-failed/216797/7 "2020-01-29T11:13:34Z")

</div>

Ok 😉  
Thanks!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 26, 2020, 11:13am UTC](https://discuss.elastic.co/t/search-phase-execution-exception-with-reason-all-shards-failed/216797/8 "2020-02-26T11:13:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.