# Courier Fetch: 33 of 893 shards failed

**URL:** <https://discuss.elastic.co/t/courier-fetch-33-of-893-shards-failed/26412>\
**Category:** Kibana\
**Created:** [July 28, 2015, 1:22pm UTC](https://discuss.elastic.co/t/courier-fetch-33-of-893-shards-failed/26412 "2015-07-28T13:22:16Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![daks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daks/32/3883_2.png) [@daks](https://discuss.elastic.co/u/daks)\
**Post date:** [July 28, 2015, 1:22pm UTC](https://discuss.elastic.co/t/courier-fetch-33-of-893-shards-failed/26412/1 "2015-07-28T13:22:16Z")

</div>

Hello,

I'm running a small ElasticSearch cluster (2 data nodes, 1 master node with no data) in which I inject logs with Logstash. I use Kibana 4.0.1.  
I got daily indexes with 3 shards and 1 replica.

All was fine until yesterday when I changed an ElasticSearch parameter. I changed index.merge.scheduler.max\_thread\_count: 1, after having read [https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html) and because my servers use spinning hard disks.

I changed this parameter on my 3 servers and since, Kibana reports the error "Courier Fetch: 33 of 893 shards failed." when I want to see recent logs.

My problem is: I got no errors on my elasticsearch logs so I don't know which indexes fails and can't try to repair them.

Anybody knows how to solve this problem, or how to find which indexes are failing?

Thanks

(by the way, I removed this new parameter on ES since I got this problem)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 28, 2015, 9:46pm UTC](https://discuss.elastic.co/t/courier-fetch-33-of-893-shards-failed/26412/2 "2015-07-28T21:46:57Z")

</div>

It means that some of your shards are not available.

Take a look at things like the `_cat/allocation`, `_cat/shards` and `_cat/recovery` APIs to see what is happening.

---

<div class="post-metadata">

**Author:** ![daks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daks/32/3883_2.png) [@daks](https://discuss.elastic.co/u/daks)\
**Post date:** [July 29, 2015, 6:50am UTC](https://discuss.elastic.co/t/courier-fetch-33-of-893-shards-failed/26412/3 "2015-07-29T06:50:07Z")

</div>

Thanks, `_cat/allocation` does not seems to reveal any interesting information.  
`_cat/shards` lists all my indexes and its shards indicating if it's a primary or a replica, thay are all at state `STARTED`.

Looking at `_cat/recovery` it seems that some shards are in the middle of a 'transit' between two nodes. (as indicated at [https://www.elastic.co/guide/en/elasticsearch/reference/1.4/cat-recovery.html](https://www.elastic.co/guide/en/elasticsearch/reference/1.4/cat-recovery.html) as I'm running 1.4). It looks like

```
index-2015.03.09 2 6154 replica done <data1> <data2> n/a n/a 194 0.5% 271085265 0.0%

```

Percent values are often 0 or 100% but some got 25.0% or 0.5%, 0.9%.

I'll try to find how to repair this.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 29, 2015, 6:54am UTC](https://discuss.elastic.co/t/courier-fetch-33-of-893-shards-failed/26412/4 "2015-07-29T06:54:18Z")

</div>

Ok it could be something else, take a look in your ES logs as well.

---

<div class="post-metadata">

**Author:** ![daks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daks/32/3883_2.png) [@daks](https://discuss.elastic.co/u/daks)\
**Post date:** [July 29, 2015, 6:56am UTC](https://discuss.elastic.co/t/courier-fetch-33-of-893-shards-failed/26412/5 "2015-07-29T06:56:57Z")

</div>

They don't reveal anything interesting. I got the same errors in the past and they were always information in ES logs but it's not the case this time.

Moreover the Kopf plugin indicates a green cluster and looking at `_cluster/health` or `_cluster/health?level=indices` indicated all indices are green, no red or yellow indication.

---

<div class="post-metadata">

**Author:** ![daks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daks/32/3883_2.png) [@daks](https://discuss.elastic.co/u/daks)\
**Post date:** [July 29, 2015, 12:08pm UTC](https://discuss.elastic.co/t/courier-fetch-33-of-893-shards-failed/26412/6 "2015-07-29T12:08:55Z")

</div>

I had a problem with my /var partition which resulted in no ES logs... After solving this problem, I found the errors in the logs about ParseFailure. I tried to repair the Lucene indexes with the correct java command but without success ('No problems were detected with this index.'). I finally decided to close/delete those indices (they were old and not needed anymore).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:15pm UTC](https://discuss.elastic.co/t/courier-fetch-33-of-893-shards-failed/26412/7 "2017-07-06T14:15:42Z")

</div>


