Courier Fetch: 1 of 610 shards failed



Could you help me to understand why I have every times the fallowing message when I try to see the dashboard: Courier Fetch: 1 of 610 shards failed.

I have only a cluster with a node.

Otherwise I don't understand why the shard size is getting high on monitoring panel.


(Mark Walkom) #2

Check your ES logs for more on that error.

What sort of data are you sending to ES, and how are you sending it?


Thanks for your reply @warkolm

My ES logs is:

at [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$ [netty-common-4.1.9.Final.jar:4.1.9.Final]
at [?:1.8.0_121]

It's Syslog data I send to ES. I use syslog directly in input plugin.


(Mark Walkom) #4

Is there more to that log?

How large is your node? How many indices? How many shards? What version of the stack are you on?


Yes @warkolm, there is another logs.

I forget to notice that I had this behavior since x-pack installation: Courier Fetch: 1 of 610 shards failed.
I have just 1 node, I create two new indexes than logstash-*. I don't change the number of shards by default which is 5. And in the monitoring panel, I can notice my shard size becoming too high 672 (I don't understand this). I use ES 5.4.0

And another logs are:

[2017-05-30T10:34:59,183][DEBUG][o.e.a.s.TransportSearchAction] [node-ircad] [ircad-2017.20.29][3], node[ZwCaCsUmRwKyyzkAimImfg], [P], s[STARTED], a[id=5oYgUNB_R6ui5__4ap0NZQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[ircad-2017.38.24, ircad-2017.21.29,

indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[], routing='null', preference='1496133297584', requestCache=null, scroll=null, source={
"size" : 0,
"query" : {
"bool" : {
"must" : [
"query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 10000,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"analyze_wildcard" : true,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0

org.elasticsearch.transport.RemoteTransportException: [node-ircad][][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportService$7@2edd31d7 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@1b862a97[Running, pool size = 7, active threads = 4, queued tasks = 1000, completed tasks = 134139]]
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution( ~[elasticsearch-5.4.0.jar:5.4.0]
at java.util.concurrent.ThreadPoolExecutor.reject( ~[?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.execute( ~[?:1.8.0_121]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute( ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute( ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.sendLocalRequest( ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.access$000( ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService$3.sendRequest( ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.sendRequestInternal( ~[elasticsearch-5.4.0.jar:5.4.0]
at ~[?:?]
at$600( ~[?:?]
at$1.sendRequest( ~[?:?]

(Mark Walkom) #6

That means your node is overloaded.

If you only have a single node, I'd say you have too many shards. You should look to use the _shrink API on any old index and reduce the shard count. If you have data that is less than ~50GB per index, then shrink them to a single shard.


Thanks @warkolm

I will try this and will inform you.

But, why the number of shard increase every day when I take a look to monitoring panel. (see below)
Initially the number of shard 5. Or, it's the different shard?

(Mark Walkom) #8

New indices get created every day, which means more shards.

(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.