Courier fetch: N of N shards failed

Elastic Stack Elasticsearch
1

Hi All,

I would like to divide 'ismb.miki.host.ram.usage" with an number in percentile

kibana_jsoninput.png1416×440 22.4 KB

I found an issue and follow that steps, and then create a scripted field as following

kibana_scripted_field.png1297×595 19 KB

Then I create a visualization by Metric, set average to aggregations and set my custom scripted filed to field, but some warning displayed as following (Courier fetch: N of N shareled)

**

kibana_visualize_metric.png1026×518 25.9 KB

To solve the problem I refer to https://github.com/elastic/kibana/issues/3221 to increase thread search queue_size in elasticsearch.yml (every node)

##### Thread pool #####

# Search pool
threadpool.search.type: fixed
threadpool.search.size: 200
threadpool.search.queue_size: 20000

My parts of shards status as

$curl -XGET -u es_admin:iiiiii http://elasticsearch:9200/_cat/shards?pretty=true
logstash-2015.08.01 0 p STARTED     6743      2mb 140.92.25.161 enode1
logstash-2015.08.01 0 r UNASSIGNED
logstash-2015.08.01 3 p STARTED     6957      2mb 140.92.25.161 enode1
logstash-2015.08.01 3 r UNASSIGNED
logstash-2015.08.01 1 p STARTED     6889      2mb 140.92.25.161 enode1
logstash-2015.08.01 1 r UNASSIGNED
logstash-2015.08.01 4 p STARTED     6861      2mb 140.92.25.161 enode1
logstash-2015.08.01 4 r UNASSIGNED
logstash-2015.08.21 2 p STARTED     2619  852.8kb 140.92.25.126 elknode
logstash-2015.08.21 2 r STARTED     2619    1.6mb 140.92.25.161 enode1
logstash-2015.08.21 0 p STARTED     2591    818kb 140.92.25.126 elknode
logstash-2015.08.21 0 r STARTED     2591  844.4kb 140.92.25.161 enode1
logstash-2015.08.21 3 p STARTED     2622  843.5kb 140.92.25.126 elknode
logstash-2015.08.21 3 r STARTED     2622    1.6mb 140.92.25.161 enode1
logstash-2015.08.21 1 p STARTED     2530  807.6kb 140.92.25.126 elknode
logstash-2015.08.21 1 r STARTED     2530    817kb 140.92.25.161 enode1
logstash-2015.08.21 4 p STARTED     2616    1.6mb 140.92.25.126 elknode
logstash-2015.08.21 4 r STARTED     2616    1.6mb 140.92.25.161 enode1
franky-2015.08.05   2 p STARTED       19   30.7kb 140.92.25.161 enode1
franky-2015.08.05   2 r UNASSIGNED
franky-2015.08.05   0 p STARTED       12   34.4kb 140.92.25.161 enode1
franky-2015.08.05   0 r UNASSIGNED
franky-2015.08.05   3 p STARTED       17   24.9kb 140.92.25.161 enode1
franky-2015.08.05   3 r UNASSIGNED
franky-2015.08.05   1 p STARTED       19   18.4kb 140.92.25.161 enode1
franky-2015.08.05   1 r UNASSIGNED
franky-2015.08.05   4 p STARTED       14   45.8kb 140.92.25.161 enode1
franky-2015.08.05   4 r UNASSIGNED
logstash-2015.07.31 4 p STARTED     6958      2mb 140.92.25.161 enode1
logstash-2015.07.31 4 r UNASSIGNED

Did I misunderstand something?? Please someone help

Jason

2

What do your ES logs show?
Are there reports of fielddata circuit breaker? Can you check to see if you have threadpool rejections?

3

Hi Mark,

ES Logs shown as following, but I don't really understand why its happend?

[2015-08-24 09:32:19,499][DEBUG][action.search.type       ] [elknode] [logstash-2015.07.21][2], node[XyqXbu_DRcm4x4pM8nAkCg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4a59ba99] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [enode1][inet[/140.92.25.161:9301]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [logstash-2015.07.21][2]: query[ConstantScore(BooleanFilter(+QueryWrapperFilter(+(ismb.miki.vm.uuid:"37de5a11 d83b 410d abac a57647215318") +_all:ismb.miki.vm.vcpu.usage +_all:ismb.miki.vm.vram.max) +cache(@timestamp:[1439949973255 TO 1440123675780])))],from[-1],size[0]: Parse Failure [Failed to parse source [{"query":{"filtered":{"query":{"query_string":{"query":"*","analyze_wildcard":true}},"filter":{"bool":{"must":[{"query":{"query_string":{"analyze_wildcard":true,"query":"ismb.miki.vm.uuid:\"37de5a11-d83b-410d-abac-a57647215318\" AND ismb.miki.vm.vcpu.usage AND ismb.miki.vm.vram.max"}}},{"range":{"@timestamp":{"gte":1439949973255,"lte":1440123675780}}}],"must_not":[]}}}},"size":0,"aggs":{"2":{"date_range":{"field":"@timestamp","ranges":[{"from":"now-60m","to":"now-50m"},{"from":"now-50m","to":"now-40m"},{"from":"now-40m","to":"now-30m"},{"from":"now-30m","to":"now-20m"},{"from":"now-20m","to":"now-10m"},{"from":"now-10m","to":"now"}]},"aggs":{"1":{"max":{"script":"doc['ismb.miki.vm.vcpu.usage'].value/100","lang":"expression"}},"3":{"max":{"script":"doc['ismb.miki.vm.vram.usage'].value/doc['ismb.miki.vm.vram.max'].value","lang":"expression"}}}}}}]]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:747)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:306)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
        at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:176)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.expression.ExpressionScriptCompilationException: Field [ismb.miki.vm.vcpu.usage] used in expression does not exist in mappings
        at org.elasticsearch.script.expression.ExpressionScriptEngineService.search(ExpressionScriptEngineService.java:129)
        at org.elasticsearch.script.ScriptService.search(ScriptService.java:484)
        at org.elasticsearch.search.aggregations.support.ValuesSourceParser.createScript(ValuesSourceParser.java:188)
        at org.elasticsearch.search.aggregations.support.ValuesSourceParser.config(ValuesSourceParser.java:148)
        at org.elasticsearch.search.aggregations.metrics.NumericValuesSourceMetricsAggregatorParser.parse(NumericValuesSourceMetricsAggregatorParser.java:65)
        at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:148)
        at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:138)
        at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:78)
        at org.elasticsearch.search.aggregations.AggregationParseElement.parse(AggregationParseElement.java:60)
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
        ... 11 more

Too long message, I split two reply

4

Reproduce Steps,

  1. There are there different documents (field) in my elasticsearch cluster, first is

    {
    "ismb.miki.timestamp": 1440382731000,
    "ismb.miki.host": "192.168.10.19",
    "ismb.miki.host.cpu.usage": 14,
    "ismb.miki.host.ram.usage": 7184532,
    "ismb.miki.host.ram.max": 32824548,
    "ismb.miki.host.uptime": 1204982
    }

second is

{
    "ismb.miki.timestamp": 1440382726000,
    "ismb.miki.vm.uuid": "515b9f57-2f71-4435-aac2-6b19da52a89f",
    "ismb.miki.host": "192.168.10.19",
    "ismb.miki.vm.vcpu.usage": 101,
    "ismb.miki.vm.vcpu.num": 1,
    "ismb.miki.vm.vram.usage": 346640,
    "ismb.miki.vm.vram.max": 262144
}

third is

{
    "ismb.miki.timestamp": 1440382726000,
    "ismb.miki.vm.uuid": "3250674f-98bd-4248-9087-c3a1c5205fb2",
    "ismb.miki.host": "192.168.10.19",
    "ismb.miki.vnic.mac": "00:16:3e:b2:b1:c7",
    "ismb.miki.vnic.rx": 377530565,
    "ismb.miki.vnic.tx": 3149227
}
  1. For monitoring usage of CPU, RAM and Network Traffic (TX RX), I created some scripted field from kibana "Settings" page, for example of monitoring CPU and Memory usage as following

scriptedfield.png1655×725 36.9 KB

scriptedfield2.png1659×699 33.6 KB

  1. Then I created visualization by "Visualize" of Kibana, following is my visualization setting, when I load this page, "Courier fetch: N of N shards failed" is always display, and previous ES log printed "failed to execute...." at the same time

scriptedfield3.png1661×979 62.3 KB

Jason

5

That's why.

6

Hi Mark,

Thanks for replying!

I think the key 'ismb.miki.vm.vcpu.usage' is actually exist in elasticsearch cluster, but I don't really understand what is "used in expression does not exist in mappings" mean?

The problem of following topic is similar as mine, is it right or any suggestion?

Jason

7

Does ismb.miki.vm.vcpu.usage exist in the logstash-2015.07.21 index?

8

Hi Mark,

You are right, "ismb.miki.vm.vcpu.usage" doesn't exist in the logstash-2015.07.21

but it makes me confused, is scripted field key must be existed in each index?

BTW, I change 'action' from DEBUG to INFO in logging.yml, but it still display 'Courier fetch' in kibana when execute scripted field query everytime

Jason