Courier Fetch: 1 of 610 shards failed


Could you help me to understand why I have every times the fallowing message when I try to see the dashboard: Courier Fetch: 1 of 610 shards failed.

I have only a cluster with a node.

Otherwise I don't understand why the shard size is getting high on monitoring panel.


Check your ES logs for more on that error.

What sort of data are you sending to ES, and how are you sending it?

Thanks for your reply @warkolm

My ES logs is:

at [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$ [netty-common-4.1.9.Final.jar:4.1.9.Final]
at [?:1.8.0_121]

It's Syslog data I send to ES. I use syslog directly in input plugin.


Is there more to that log?

How large is your node? How many indices? How many shards? What version of the stack are you on?

Yes @warkolm, there is another logs.

I forget to notice that I had this behavior since x-pack installation: Courier Fetch: 1 of 610 shards failed.
I have just 1 node, I create two new indexes than logstash-*. I don't change the number of shards by default which is 5. And in the monitoring panel, I can notice my shard size becoming too high 672 (I don't understand this). I use ES 5.4.0

And another logs are:

[2017-05-30T10:34:59,183][DEBUG][o.e.a.s.TransportSearchAction] [node-ircad] [ircad-2017.20.29][3], node[ZwCaCsUmRwKyyzkAimImfg], [P], s[STARTED], a[id=5oYgUNB_R6ui5__4ap0NZQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[ircad-2017.38.24, ircad-2017.21.29,

indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[], routing='null', preference='1496133297584', requestCache=null, scroll=null, source={
"size" : 0,
"query" : {
"bool" : {
"must" : [
"query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 10000,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"analyze_wildcard" : true,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0

org.elasticsearch.transport.RemoteTransportException: [node-ircad][][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportService$7@2edd31d7 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@1b862a97[Running, pool size = 7, active threads = 4, queued tasks = 1000, completed tasks = 134139]]
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution( ~[elasticsearch-5.4.0.jar:5.4.0]
at java.util.concurrent.ThreadPoolExecutor.reject( ~[?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.execute( ~[?:1.8.0_121]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute( ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute( ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.sendLocalRequest( ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.access$000( ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService$3.sendRequest( ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.sendRequestInternal( ~[elasticsearch-5.4.0.jar:5.4.0]
at ~[?:?]
at$600( ~[?:?]
at$1.sendRequest( ~[?:?]

That means your node is overloaded.

If you only have a single node, I'd say you have too many shards. You should look to use the _shrink API on any old index and reduce the shard count. If you have data that is less than ~50GB per index, then shrink them to a single shard.

Thanks @warkolm

I will try this and will inform you.

But, why the number of shard increase every day when I take a look to monitoring panel. (see below)
Initially the number of shard 5. Or, it's the different shard?

New indices get created every day, which means more shards.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.