# Courier Fetch: N of N shards failed in kibana

**URL:** <https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed-in-kibana/124498>\
**Category:** Elasticsearch\
**Created:** [March 19, 2018, 8:04am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed-in-kibana/124498 "2018-03-19T08:04:32Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![shwesinhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwesinhan/32/78663_2.png) [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\
**Post date:** [March 19, 2018, 8:04am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed-in-kibana/124498/1 "2018-03-19T08:04:32Z")

</div>

hi there.

i run elk since 6 months ago. it is being stable.

but these days i got this msg very often

> courier Fetch: 23 of 420 shards failed.

could anybody explain on this msg?

i have no idea how to fix.

my scenario is that `app logs streaming > logstash > aws elastic search domain > kibana ui`

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 19, 2018, 8:06am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed-in-kibana/124498/2 "2018-03-19T08:06:41Z")

</div>

What is the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html)?

---

<div class="post-metadata">

**Author:** ![shwesinhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwesinhan/32/78663_2.png) [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\
**Post date:** [March 19, 2018, 8:10am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed-in-kibana/124498/3 "2018-03-19T08:10:26Z")

</div>

here it is @Christian_Dahlqvist

> {  
> "\_nodes": {  
> "total": 1,  
> "successful": 1,  
> "failed": 0  
> },  
> "cluster\_name": "891349355538:pgwdev",  
> "timestamp": 1521446981178,  
> "status": "yellow",  
> "indices": {  
> "count": 88,  
> "shards": {  
> "total": 432,  
> "primaries": 432,  
> "replication": 0,  
> "index": {  
> "shards": {  
> "min": 1,  
> "max": 5,  
> "avg": 4.909090909090909  
> },  
> "primaries": {  
> "min": 1,  
> "max": 5,  
> "avg": 4.909090909090909  
> },  
> "replication": {  
> "min": 0,  
> "max": 0,  
> "avg": 0  
> }  
> }  
> },  
> "docs": {  
> "count": 5390254,  
> "deleted": 1  
> },  
> "store": {  
> "size": "3.9gb",  
> "size\_in\_bytes": 4210638451,  
> "throttle\_time": "0s",  
> "throttle\_time\_in\_millis": 0  
> },  
> "fielddata": {  
> "memory\_size": "845.4kb",  
> "memory\_size\_in\_bytes": 865744,  
> "evictions": 0  
> },  
> "query\_cache": {  
> "memory\_size": "4.4mb",  
> "memory\_size\_in\_bytes": 4686202,  
> "total\_count": 1546397,  
> "hit\_count": 889013,  
> "miss\_count": 657384,  
> "cache\_size": 36817,  
> "cache\_count": 36821,  
> "evictions": 4  
> },  
> "completion": {  
> "size": "0b",  
> "size\_in\_bytes": 0  
> },  
> "segments": {  
> "count": 2292,  
> "memory": "31.3mb",  
> "memory\_in\_bytes": 32861595,  
> "terms\_memory": "27.6mb",  
> "terms\_memory\_in\_bytes": 29018539,  
> "stored\_fields\_memory": "1.6mb",  
> "stored\_fields\_memory\_in\_bytes": 1768592,  
> "term\_vectors\_memory": "936b",  
> "term\_vectors\_memory\_in\_bytes": 936,  
> "norms\_memory": "29.3kb",  
> "norms\_memory\_in\_bytes": 30016,  
> "points\_memory": "65.5kb",  
> "points\_memory\_in\_bytes": 67168,  
> "doc\_values\_memory": "1.8mb",  
> "doc\_values\_memory\_in\_bytes": 1976344,  
> "index\_writer\_memory": "0b",  
> "index\_writer\_memory\_in\_bytes": 0,  
> "version\_map\_memory": "0b",  
> "version\_map\_memory\_in\_bytes": 0,  
> "fixed\_bit\_set": "0b",  
> "fixed\_bit\_set\_memory\_in\_bytes": 0,  
> "max\_unsafe\_auto\_id\_timestamp": 1520593756131,  
> "file\_sizes": {}  
> }  
> },  
> "nodes": {  
> "count": {  
> "total": 1,  
> "data": 1,  
> "coordinating\_only": 0,  
> "master": 1,  
> "ingest": 1  
> },  
> "versions": [  
> "5.5.2"  
> ],  
> "os": {  
> "available\_processors": 1,  
> "allocated\_processors": 1,  
> "names": [  
> {  
> "count": 1  
> }  
> ],  
> "mem": {  
> "total": "1.9gb",  
> "total\_in\_bytes": 2093498368,  
> "free": "141.2mb",  
> "free\_in\_bytes": 148090880,  
> "used": "1.8gb",  
> "used\_in\_bytes": 1945407488,  
> "free\_percent": 7,  
> "used\_percent": 93  
> }  
> },  
> "process": {  
> "cpu": {  
> "percent": 2  
> },  
> "open\_file\_descriptors": {  
> "min": 1412,  
> "max": 1412,  
> "avg": 1412  
> }  
> },  
> "jvm": {  
> "max\_uptime": "9.8d",  
> "max\_uptime\_in\_millis": 853696503,  
> "mem": {  
> "heap\_used": "434.7mb",  
> "heap\_used\_in\_bytes": 455873608,  
> "heap\_max": "1015.6mb",  
> "heap\_max\_in\_bytes": 1065025536  
> },  
> "threads": 113  
> },  
> "fs": {  
> "total": "11.6gb",  
> "total\_in\_bytes": 12548489216,  
> "free": "7.6gb",  
> "free\_in\_bytes": 8254406656,  
> "available": "7gb",  
> "available\_in\_bytes": 7593385984  
> },  
> "network\_types": {  
> "transport\_types": {  
> "netty4": 1  
> },  
> "http\_types": {  
> "filter-jetty": 1  
> }  
> }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 19, 2018, 8:13am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed-in-kibana/124498/4 "2018-03-19T08:13:26Z")

</div>

That is a lot of shards given the amount off heap you have available in your cluster. Read [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) for some guidance on how large you shards should be and how many you should aim to have in your cluster.

You can use the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-shrink-index.html) to reduce the shard count by some margin. If you need to go further, you may need to use the [reindex API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docs-reindex.html) to reindex your data into e.g. monthly indices instead.

---

<div class="post-metadata">

**Author:** ![shwesinhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwesinhan/32/78663_2.png) [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\
**Post date:** [March 26, 2018, 8:21am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed-in-kibana/124498/5 "2018-03-26T08:21:57Z")

</div>

if i use reindex API, there will be duplication problem??  
and using monthly indices still same performance with daily indices??

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 26, 2018, 8:29am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed-in-kibana/124498/6 "2018-03-26T08:29:47Z")

</div>

Given how little data you have in the cluster, I don;'t see monthly indices getting very large, so I would expect them to perform much better. As seen in the blog post I linked to we often recommend shard sizes in the tens of GB, which you seem unlikely to reach even with monthly indices.

While reindexing is going on, you could end up with data being duplicated as the monthly index would potentially also match the index pattern. Once the reindexing has completed I would however expect the daily indices to be deleted. As you have relatively little data in your cluster I would not necessarily expect reindexing to take very long.

If this is not acceptable, you can reindex into an index that does not match the daily index pattern and then instead create an alias for the index at the time you delete the daily indices. This will reduce the amount off time duplicates can be seen in the system.

---

<div class="post-metadata">

**Author:** ![shwesinhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shwesinhan/32/78663_2.png) [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\
**Post date:** [March 26, 2018, 8:51am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed-in-kibana/124498/7 "2018-03-26T08:51:16Z")

</div>

@Christian_Dahlqvist thanks for your reply and advise.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 23, 2018, 8:51am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed-in-kibana/124498/8 "2018-04-23T08:51:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.