Courier Fetch: 1 of 610 shards failed

[Baco]

Hello,

Could you help me to understand why I have every times the fallowing message when I try to see the dashboard: Courier Fetch: 1 of 610 shards failed.

I have only a cluster with a node.

Otherwise I don't understand why the shard size is getting high on monitoring panel.

Thanks

[warkolm (Mark Walkom)]

Check your ES logs for more on that error.

What sort of data are you sending to ES, and how are you sending it?

[Baco]

Thanks for your reply @warkolm

My ES logs is:

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.9.Final.jar:4.1.9.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

It's Syslog data I send to ES. I use syslog directly in input plugin.

Thanks

[warkolm (Mark Walkom)]

Is there more to that log?

How large is your node? How many indices? How many shards? What version of the stack are you on?

[Baco]

Yes @warkolm, there is another logs.

I forget to notice that I had this behavior since x-pack installation: Courier Fetch: 1 of 610 shards failed.
I have just 1 node, I create two new indexes than logstash-*. I don't change the number of shards by default which is 5. And in the monitoring panel, I can notice my shard size becoming too high 672 (I don't understand this). I use ES 5.4.0

And another logs are:

[2017-05-30T10:34:59,183][DEBUG][o.e.a.s.TransportSearchAction] [node-ircad] [ircad-2017.20.29][3], node[ZwCaCsUmRwKyyzkAimImfg], [P], s[STARTED], a[id=5oYgUNB_R6ui5__4ap0NZQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[ircad-2017.38.24, ircad-2017.21.29,

indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[], routing='null', preference='1496133297584', requestCache=null, scroll=null, source={
"size" : 0,
"query" : {
"bool" : {
"must" : [
{
"query_string" : {
"query" : "*",
"fields" : [ ],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "or",
"auto_generate_phrase_queries" : false,
"max_determinized_states" : 10000,
"enable_position_increments" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"analyze_wildcard" : true,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
}
},
{

org.elasticsearch.transport.RemoteTransportException: [node-ircad][192.168.228.96:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportService$7@2edd31d7 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@1b862a97[Running, pool size = 7, active threads = 4, queued tasks = 1000, completed tasks = 134139]]
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:50) ~[elasticsearch-5.4.0.jar:5.4.0]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) ~[?:1.8.0_121]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:623) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.access$000(TransportService.java:73) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:133) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:569) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.sendWithUser(SecurityServerTransportInterceptor.java:146) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor.access$600(SecurityServerTransportInterceptor.java:63) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:128) ~[?:?]

[warkolm (Mark Walkom)]

@Baco

queue capacity = 1000

That means your node is overloaded.

If you only have a single node, I'd say you have too many shards. You should look to use the _shrink API on any old index and reduce the shard count. If you have data that is less than ~50GB per index, then shrink them to a single shard.

[Baco]

Thanks @warkolm

I will try this and will inform you.

But, why the number of shard increase every day when I take a look to monitoring panel. (see below)
Initially the number of shard 5. Or, it's the different shard?

[warkolm (Mark Walkom)]

New indices get created every day, which means more shards.