Courier fetch: N of N shards failed

(Jason Zheng) #1

Hi All,

I would like to divide '" with an number in percentile

I found an issue and follow that steps, and then create a scripted field as following

Then I create a visualization by Metric, set average to aggregations and set my custom scripted filed to field, but some warning displayed as following (Courier fetch: N of N shareled)


To solve the problem I refer to to increase thread search queue_size in elasticsearch.yml (every node)

##### Thread pool #####

# Search pool fixed 200 20000

My parts of shards status as

$curl -XGET -u es_admin:iiiiii http://elasticsearch:9200/_cat/shards?pretty=true
logstash-2015.08.01 0 p STARTED     6743      2mb enode1
logstash-2015.08.01 0 r UNASSIGNED
logstash-2015.08.01 3 p STARTED     6957      2mb enode1
logstash-2015.08.01 3 r UNASSIGNED
logstash-2015.08.01 1 p STARTED     6889      2mb enode1
logstash-2015.08.01 1 r UNASSIGNED
logstash-2015.08.01 4 p STARTED     6861      2mb enode1
logstash-2015.08.01 4 r UNASSIGNED
logstash-2015.08.21 2 p STARTED     2619  852.8kb elknode
logstash-2015.08.21 2 r STARTED     2619    1.6mb enode1
logstash-2015.08.21 0 p STARTED     2591    818kb elknode
logstash-2015.08.21 0 r STARTED     2591  844.4kb enode1
logstash-2015.08.21 3 p STARTED     2622  843.5kb elknode
logstash-2015.08.21 3 r STARTED     2622    1.6mb enode1
logstash-2015.08.21 1 p STARTED     2530  807.6kb elknode
logstash-2015.08.21 1 r STARTED     2530    817kb enode1
logstash-2015.08.21 4 p STARTED     2616    1.6mb elknode
logstash-2015.08.21 4 r STARTED     2616    1.6mb enode1
franky-2015.08.05   2 p STARTED       19   30.7kb enode1
franky-2015.08.05   2 r UNASSIGNED
franky-2015.08.05   0 p STARTED       12   34.4kb enode1
franky-2015.08.05   0 r UNASSIGNED
franky-2015.08.05   3 p STARTED       17   24.9kb enode1
franky-2015.08.05   3 r UNASSIGNED
franky-2015.08.05   1 p STARTED       19   18.4kb enode1
franky-2015.08.05   1 r UNASSIGNED
franky-2015.08.05   4 p STARTED       14   45.8kb enode1
franky-2015.08.05   4 r UNASSIGNED
logstash-2015.07.31 4 p STARTED     6958      2mb enode1
logstash-2015.07.31 4 r UNASSIGNED

Did I misunderstand something?? Please someone help


(Mark Walkom) #2

What do your ES logs show?
Are there reports of fielddata circuit breaker? Can you check to see if you have threadpool rejections?

(Jason Zheng) #3

Hi Mark,

ES Logs shown as following, but I don't really understand why its happend?

[2015-08-24 09:32:19,499][DEBUG][       ] [elknode] [logstash-2015.07.21][2], node[XyqXbu_DRcm4x4pM8nAkCg], [P], s[STARTED]: Failed to execute [] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [enode1][inet[/]][indices:data/read/search[phase/query]]
Caused by: [logstash-2015.07.21][2]: query[ConstantScore(BooleanFilter(+QueryWrapperFilter(+(ismb.miki.vm.uuid:"37de5a11 d83b 410d abac a57647215318") +_all:ismb.miki.vm.vcpu.usage +_all:ismb.miki.vm.vram.max) +cache(@timestamp:[1439949973255 TO 1440123675780])))],from[-1],size[0]: Parse Failure [Failed to parse source [{"query":{"filtered":{"query":{"query_string":{"query":"*","analyze_wildcard":true}},"filter":{"bool":{"must":[{"query":{"query_string":{"analyze_wildcard":true,"query":"ismb.miki.vm.uuid:\"37de5a11-d83b-410d-abac-a57647215318\" AND ismb.miki.vm.vcpu.usage AND ismb.miki.vm.vram.max"}}},{"range":{"@timestamp":{"gte":1439949973255,"lte":1440123675780}}}],"must_not":[]}}}},"size":0,"aggs":{"2":{"date_range":{"field":"@timestamp","ranges":[{"from":"now-60m","to":"now-50m"},{"from":"now-50m","to":"now-40m"},{"from":"now-40m","to":"now-30m"},{"from":"now-30m","to":"now-20m"},{"from":"now-20m","to":"now-10m"},{"from":"now-10m","to":"now"}]},"aggs":{"1":{"max":{"script":"doc['ismb.miki.vm.vcpu.usage'].value/100","lang":"expression"}},"3":{"max":{"script":"doc['ismb.miki.vm.vram.usage'].value/doc['ismb.miki.vm.vram.max'].value","lang":"expression"}}}}}}]]
        at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
        at java.util.concurrent.ThreadPoolExecutor$
Caused by: org.elasticsearch.script.expression.ExpressionScriptCompilationException: Field [ismb.miki.vm.vcpu.usage] used in expression does not exist in mappings
        ... 11 more

Too long message, I split two reply

(Jason Zheng) #4

Reproduce Steps,

  1. There are there different documents (field) in my elasticsearch cluster, first is

    "ismb.miki.timestamp": 1440382731000,
    "": "",
    "": 14,
    "": 7184532,
    "": 32824548,
    "": 1204982

second is

    "ismb.miki.timestamp": 1440382726000,
    "ismb.miki.vm.uuid": "515b9f57-2f71-4435-aac2-6b19da52a89f",
    "": "",
    "ismb.miki.vm.vcpu.usage": 101,
    "ismb.miki.vm.vcpu.num": 1,
    "ismb.miki.vm.vram.usage": 346640,
    "ismb.miki.vm.vram.max": 262144

third is

    "ismb.miki.timestamp": 1440382726000,
    "ismb.miki.vm.uuid": "3250674f-98bd-4248-9087-c3a1c5205fb2",
    "": "",
    "ismb.miki.vnic.mac": "00:16:3e:b2:b1:c7",
    "ismb.miki.vnic.rx": 377530565,
    "ismb.miki.vnic.tx": 3149227
  1. For monitoring usage of CPU, RAM and Network Traffic (TX RX), I created some scripted field from kibana "Settings" page, for example of monitoring CPU and Memory usage as following

  1. Then I created visualization by "Visualize" of Kibana, following is my visualization setting, when I load this page, "Courier fetch: N of N shards failed" is always display, and previous ES log printed "failed to execute...." at the same time


(Mark Walkom) #5

That's why.

(Jason Zheng) #6

Hi Mark,

Thanks for replying!

I think the key 'ismb.miki.vm.vcpu.usage' is actually exist in elasticsearch cluster, but I don't really understand what is "used in expression does not exist in mappings" mean?

The problem of following topic is similar as mine, is it right or any suggestion?


(Mark Walkom) #7

Does ismb.miki.vm.vcpu.usage exist in the logstash-2015.07.21 index?

Error while querying elasticsearch
(Jason Zheng) #8

Hi Mark,

You are right, "ismb.miki.vm.vcpu.usage" doesn't exist in the logstash-2015.07.21

but it makes me confused, is scripted field key must be existed in each index?

BTW, I change 'action' from DEBUG to INFO in logging.yml, but it still display 'Courier fetch' in kibana when execute scripted field query everytime


(system) #9