# Courier fetch: N of N shards failed

**URL:** <https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814>\
**Category:** Elasticsearch\
**Created:** [August 21, 2015, 6:03am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814 "2015-08-21T06:03:00Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jason\_Zheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_zheng/32/4041_2.png) [@Jason\_Zheng](https://discuss.elastic.co/u/Jason_Zheng)\
**Post date:** [August 21, 2015, 6:03am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814/1 "2015-08-21T06:03:00Z")

</div>

Hi All,

I would like to divide 'ismb.miki.host.ram.usage" with an number in percentile

 ![](https://us1.discourse-cdn.com/elastic/original/2X/4/43142de5b96f95979c2bc9a3a5819f8c05569668.png)

I found an issue and follow that steps, and then create a scripted field as following

 ![](https://us1.discourse-cdn.com/elastic/original/2X/0/0927c45e35ba66818380e73f30fa08fd61b317b5.png)

Then I create a visualization by Metric, set average to aggregations and set my custom scripted filed to field, but some warning displayed as following **(Courier fetch: N of N shareled)**

\*\*

 ![](https://us1.discourse-cdn.com/elastic/original/2X/a/a9559e9668bdd434be65e0f800d63d0874e87129.png)

To solve the problem I refer to [https://github.com/elastic/kibana/issues/3221](https://github.com/elastic/kibana/issues/3221) to increase thread search queue\_size in elasticsearch.yml (every node)

```
##### Thread pool #####

# Search pool
threadpool.search.type: fixed
threadpool.search.size: 200
threadpool.search.queue_size: 20000

```

My parts of shards status as

```
$curl -XGET -u es_admin:iiiiii http://elasticsearch:9200/_cat/shards?pretty=true
logstash-2015.08.01 0 p STARTED 6743 2mb 140.92.25.161 enode1
logstash-2015.08.01 0 r UNASSIGNED
logstash-2015.08.01 3 p STARTED 6957 2mb 140.92.25.161 enode1
logstash-2015.08.01 3 r UNASSIGNED
logstash-2015.08.01 1 p STARTED 6889 2mb 140.92.25.161 enode1
logstash-2015.08.01 1 r UNASSIGNED
logstash-2015.08.01 4 p STARTED 6861 2mb 140.92.25.161 enode1
logstash-2015.08.01 4 r UNASSIGNED
logstash-2015.08.21 2 p STARTED 2619 852.8kb 140.92.25.126 elknode
logstash-2015.08.21 2 r STARTED 2619 1.6mb 140.92.25.161 enode1
logstash-2015.08.21 0 p STARTED 2591 818kb 140.92.25.126 elknode
logstash-2015.08.21 0 r STARTED 2591 844.4kb 140.92.25.161 enode1
logstash-2015.08.21 3 p STARTED 2622 843.5kb 140.92.25.126 elknode
logstash-2015.08.21 3 r STARTED 2622 1.6mb 140.92.25.161 enode1
logstash-2015.08.21 1 p STARTED 2530 807.6kb 140.92.25.126 elknode
logstash-2015.08.21 1 r STARTED 2530 817kb 140.92.25.161 enode1
logstash-2015.08.21 4 p STARTED 2616 1.6mb 140.92.25.126 elknode
logstash-2015.08.21 4 r STARTED 2616 1.6mb 140.92.25.161 enode1
franky-2015.08.05 2 p STARTED 19 30.7kb 140.92.25.161 enode1
franky-2015.08.05 2 r UNASSIGNED
franky-2015.08.05 0 p STARTED 12 34.4kb 140.92.25.161 enode1
franky-2015.08.05 0 r UNASSIGNED
franky-2015.08.05 3 p STARTED 17 24.9kb 140.92.25.161 enode1
franky-2015.08.05 3 r UNASSIGNED
franky-2015.08.05 1 p STARTED 19 18.4kb 140.92.25.161 enode1
franky-2015.08.05 1 r UNASSIGNED
franky-2015.08.05 4 p STARTED 14 45.8kb 140.92.25.161 enode1
franky-2015.08.05 4 r UNASSIGNED
logstash-2015.07.31 4 p STARTED 6958 2mb 140.92.25.161 enode1
logstash-2015.07.31 4 r UNASSIGNED

```

Did I misunderstand something?? Please someone help

Jason

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 22, 2015, 8:40am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814/2 "2015-08-22T08:40:26Z")

</div>

What do your ES logs show?  
Are there reports of fielddata circuit breaker? Can you check to see if you have threadpool rejections?

---

<div class="post-metadata">

**Author:** ![Jason\_Zheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_zheng/32/4041_2.png) [@Jason\_Zheng](https://discuss.elastic.co/u/Jason_Zheng)\
**Post date:** [August 24, 2015, 2:59am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814/3 "2015-08-24T02:59:52Z")

</div>

Hi Mark,

ES Logs shown as following, but I don't really understand why its happend?

```
[2015-08-24 09:32:19,499][DEBUG][action.search.type] [elknode] [logstash-2015.07.21][2], node[XyqXbu_DRcm4x4pM8nAkCg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4a59ba99] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [enode1][inet[/140.92.25.161:9301]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [logstash-2015.07.21][2]: query[ConstantScore(BooleanFilter(+QueryWrapperFilter(+(ismb.miki.vm.uuid:"37de5a11 d83b 410d abac a57647215318") +_all:ismb.miki.vm.vcpu.usage +_all:ismb.miki.vm.vram.max) +cache(@timestamp:[1439949973255 TO 1440123675780])))],from[-1],size[0]: Parse Failure [Failed to parse source [{"query":{"filtered":{"query":{"query_string":{"query":"*","analyze_wildcard":true}},"filter":{"bool":{"must":[{"query":{"query_string":{"analyze_wildcard":true,"query":"ismb.miki.vm.uuid:\"37de5a11-d83b-410d-abac-a57647215318\" AND ismb.miki.vm.vcpu.usage AND ismb.miki.vm.vram.max"}}},{"range":{"@timestamp":{"gte":1439949973255,"lte":1440123675780}}}],"must_not":[]}}}},"size":0,"aggs":{"2":{"date_range":{"field":"@timestamp","ranges":[{"from":"now-60m","to":"now-50m"},{"from":"now-50m","to":"now-40m"},{"from":"now-40m","to":"now-30m"},{"from":"now-30m","to":"now-20m"},{"from":"now-20m","to":"now-10m"},{"from":"now-10m","to":"now"}]},"aggs":{"1":{"max":{"script":"doc['ismb.miki.vm.vcpu.usage'].value/100","lang":"expression"}},"3":{"max":{"script":"doc['ismb.miki.vm.vram.usage'].value/doc['ismb.miki.vm.vram.max'].value","lang":"expression"}}}}}}]]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:747)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:306)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
        at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:176)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.expression.ExpressionScriptCompilationException: Field [ismb.miki.vm.vcpu.usage] used in expression does not exist in mappings
        at org.elasticsearch.script.expression.ExpressionScriptEngineService.search(ExpressionScriptEngineService.java:129)
        at org.elasticsearch.script.ScriptService.search(ScriptService.java:484)
        at org.elasticsearch.search.aggregations.support.ValuesSourceParser.createScript(ValuesSourceParser.java:188)
        at org.elasticsearch.search.aggregations.support.ValuesSourceParser.config(ValuesSourceParser.java:148)
        at org.elasticsearch.search.aggregations.metrics.NumericValuesSourceMetricsAggregatorParser.parse(NumericValuesSourceMetricsAggregatorParser.java:65)
        at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:148)
        at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:138)
        at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:78)
        at org.elasticsearch.search.aggregations.AggregationParseElement.parse(AggregationParseElement.java:60)
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
        ... 11 more

```

Too long message, I split two reply

---

<div class="post-metadata">

**Author:** ![Jason\_Zheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_zheng/32/4041_2.png) [@Jason\_Zheng](https://discuss.elastic.co/u/Jason_Zheng)\
**Post date:** [August 24, 2015, 3:00am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814/4 "2015-08-24T03:00:04Z")

</div>

Reproduce Steps,

1. There are there different documents (field) in my elasticsearch cluster, first is

second is

```
{
    "ismb.miki.timestamp": 1440382726000,
    "ismb.miki.vm.uuid": "515b9f57-2f71-4435-aac2-6b19da52a89f",
    "ismb.miki.host": "192.168.10.19",
    "ismb.miki.vm.vcpu.usage": 101,
    "ismb.miki.vm.vcpu.num": 1,
    "ismb.miki.vm.vram.usage": 346640,
    "ismb.miki.vm.vram.max": 262144
}

```

third is

```
{
    "ismb.miki.timestamp": 1440382726000,
    "ismb.miki.vm.uuid": "3250674f-98bd-4248-9087-c3a1c5205fb2",
    "ismb.miki.host": "192.168.10.19",
    "ismb.miki.vnic.mac": "00:16:3e:b2:b1:c7",
    "ismb.miki.vnic.rx": 377530565,
    "ismb.miki.vnic.tx": 3149227
}

```

1. For monitoring usage of CPU, RAM and Network Traffic (TX RX), I created some scripted field from kibana "Settings" page, for example of monitoring CPU and Memory usage as following

 ![](https://us1.discourse-cdn.com/elastic/original/2X/2/2effbed0c573dbfa09f09014daec52317e87eca9.png)

 ![](https://us1.discourse-cdn.com/elastic/original/2X/9/90ffa73d14e7350d55bbabb1dc1f1c8fa5f3635e.png)

1. Then I created visualization by "Visualize" of Kibana, following is my visualization setting, when I load this page, "Courier fetch: N of N shards failed" is always display, and previous ES log printed "failed to execute...." at the same time

 ![](https://us1.discourse-cdn.com/elastic/original/2X/7/731513b1ea2b905b99717c06bc6a9fe810b3149f.png)

Jason

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 24, 2015, 3:18am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814/5 "2015-08-24T03:18:57Z")

</div>

> [@Jason\_Zheng](#):
>
> Field [ismb.miki.vm.vcpu.usage] used in expression does not exist in mappings

That's why.

---

<div class="post-metadata">

**Author:** ![Jason\_Zheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_zheng/32/4041_2.png) [@Jason\_Zheng](https://discuss.elastic.co/u/Jason_Zheng)\
**Post date:** [August 24, 2015, 3:51am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814/6 "2015-08-24T03:51:29Z")

</div>

Hi Mark,

Thanks for replying!

I think the key 'ismb.miki.vm.vcpu.usage' is actually exist in elasticsearch cluster, but I don't really understand what is "used in expression does not exist in mappings" mean?

The problem of following topic is similar as mine, is it right or any suggestion?

> <https://github.com/elastic/elasticsearch/issues/10967>

Jason

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 24, 2015, 3:52am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814/7 "2015-08-24T03:52:56Z")

</div>

Does `ismb.miki.vm.vcpu.usage` exist in the `logstash-2015.07.21` index?

---

<div class="post-metadata">

**Author:** ![Jason\_Zheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jason_zheng/32/4041_2.png) [@Jason\_Zheng](https://discuss.elastic.co/u/Jason_Zheng)\
**Post date:** [August 24, 2015, 4:02am UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814/8 "2015-08-24T04:02:19Z")

</div>

> [@warkolm](#):
>
> logstash-2015.07.21

Hi Mark,

You are right, "ismb.miki.vm.vcpu.usage" doesn't exist in the logstash-2015.07.21

but it makes me confused, is scripted field key must be existed in each index?

BTW, I change 'action' from DEBUG to INFO in logging.yml, but it still display 'Courier fetch' in kibana when execute scripted field query everytime

Jason

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:54pm UTC](https://discuss.elastic.co/t/courier-fetch-n-of-n-shards-failed/27814/9 "2017-07-05T23:54:18Z")

</div>