# Courier Fetch: X of X shards failed

**URL:** <https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427>\
**Category:** Kibana\
**Created:** [August 26, 2017, 2:57am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427 "2017-08-26T02:57:11Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![truongdqse03303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/truongdqse03303/32/22605_2.png) [@truongdqse03303](https://discuss.elastic.co/u/truongdqse03303)\
**Post date:** [August 26, 2017, 2:57am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/1 "2017-08-26T02:57:11Z")

</div>

Hi experts.  
I use scripted field with my filebeat at field sometime have value sometime no so my scripted is:

> if (!doc['Time'].empty) {  
> return doc['Time'].date.hourOfDay + 7;  
> }

after that my filebeat-\* index not working and it throw Courier Fetch: X of X shards failed  
Please help me with this

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 26, 2017, 4:59am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/2 "2017-08-26T04:59:55Z")

</div>

There should be something in the Elasticsearch logs as well at this time?

---

<div class="post-metadata">

**Author:** ![truongdqse03303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/truongdqse03303/32/22605_2.png) [@truongdqse03303](https://discuss.elastic.co/u/truongdqse03303)\
**Post date:** [August 28, 2017, 4:32am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/3 "2017-08-28T04:32:59Z")

</div>

Hi @warkolm  
Sr for late reply  
this is my elasticsearch logs

> ```
> at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:422) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:393) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:390) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) [elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.2.jar:5.5.2]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
> Caused by: java.lang.IllegalArgumentException: Illegal list shortcut value [date].
> at org.elasticsearch.painless.Def.lookupGetter(Def.java:454) ~[?:?]
> at org.elasticsearch.painless.DefBootstrap$PIC.lookup(DefBootstrap.java:149) ~[?:?]
> at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:203) ~[?:?]
> at org.elasticsearch.painless.PainlessScript$Script.execute(if (!doc['Time'].empty) {
> return doc['Time'].date.hourOfDay + 7;
> } return null;:45) ~[?:?]
> at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:133) ~[?:?]
> at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:51) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:164) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:422) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:393) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:390) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.5.2.jar:5.5.2]
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.2.jar:5.5.2]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_141]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_141]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_141]
> 
> ```

---

<div class="post-metadata">

**Author:** ![truongdqse03303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/truongdqse03303/32/22605_2.png) [@truongdqse03303](https://discuss.elastic.co/u/truongdqse03303)\
**Post date:** [August 29, 2017, 3:45am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/4 "2017-08-29T03:45:45Z")

</div>

Hi @warkolm  
I think that error come from the logstash filter, because when i add if condition with tag, the file Time of filebeat become string type, not date type. How can i keep the data type when use if condition.  
Here is my logstash filter

```
 if "node" in [tags] {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:Time} %{WORD:Method} %{NUMBER:Http_stt} %{NUMBER:ResTime}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
    remove_field => "message"
    }
    syslog_pri { }
    date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
  }
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 29, 2017, 7:51am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/5 "2017-08-29T07:51:03Z")

</div>

It's hard to say without seeing the full error.

---

<div class="post-metadata">

**Author:** ![truongdqse03303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/truongdqse03303/32/22605_2.png) [@truongdqse03303](https://discuss.elastic.co/u/truongdqse03303)\
**Post date:** [August 29, 2017, 8:05am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/6 "2017-08-29T08:05:53Z")

</div>

I said that because i check the field Time filtered type is String so the paintless scripts not work but when i use paintless script for @timestamp, it's working without any error

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 29, 2017, 8:11am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/7 "2017-08-29T08:11:36Z")

</div>

Oh, then yes that makes sense.

---

<div class="post-metadata">

**Author:** ![truongdqse03303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/truongdqse03303/32/22605_2.png) [@truongdqse03303](https://discuss.elastic.co/u/truongdqse03303)\
**Post date:** [August 29, 2017, 8:14am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/8 "2017-08-29T08:14:21Z")

</div>

@warkolm so can you tell my why when i added the [tags] condition, all my field become string type? even the grokfilter is %{TIMESTAMP\_ISO8601}  
Thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 29, 2017, 8:15am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/9 "2017-08-29T08:15:40Z")

</div>

Probably because you are doing a date filter on a different field, `syslog_timestamp` and not `Time` as per the grok.

---

<div class="post-metadata">

**Author:** ![truongdqse03303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/truongdqse03303/32/22605_2.png) [@truongdqse03303](https://discuss.elastic.co/u/truongdqse03303)\
**Post date:** [August 29, 2017, 8:22am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/10 "2017-08-29T08:22:13Z")

</div>

Here is my grokfilter

```
 if "node" in [tags] {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:Time} %{WORD:Method} %{NUMBER:Http_stt} %{NUMBER:ResTime}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
    remove_field => "message"
    }
    syslog_pri { }
    date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
  }

```

but the type of `Time` field is string. It's because of the line ?

```
date {
          match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
        }

```

Thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 29, 2017, 8:46am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/11 "2017-08-29T08:46:34Z")

</div>

Yeah, but `syslog_timestamp` doesn't exist in your grok.

---

<div class="post-metadata">

**Author:** ![truongdqse03303](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/truongdqse03303/32/22605_2.png) [@truongdqse03303](https://discuss.elastic.co/u/truongdqse03303)\
**Post date:** [August 29, 2017, 8:51am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/12 "2017-08-29T08:51:38Z")

</div>

Hi @warkolm  
so instead of using

> match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]

i should use

> date {  
> match =\> ["Time", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
> }

? or just remove the date filter  
Thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 29, 2017, 8:59am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/13 "2017-08-29T08:59:48Z")

</div>

Just update the date filter and it should be better.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 26, 2017, 8:59am UTC](https://discuss.elastic.co/t/courier-fetch-x-of-x-shards-failed/98427/14 "2017-09-26T08:59:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.