Courier Fetch: X of X shards failed

Elastic Stack Kibana
1

Hi experts.
I use scripted field with my filebeat at field sometime have value sometime no so my scripted is:

if (!doc['Time'].empty) {
return doc['Time'].date.hourOfDay + 7;
}

after that my filebeat-* index not working and it throw Courier Fetch: X of X shards failed
Please help me with this

2

There should be something in the Elasticsearch logs as well at this time?

3

Hi @warkolm
Sr for late reply
this is my elasticsearch logs

at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:422) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:393) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:390) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) [elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.2.jar:5.5.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.lang.IllegalArgumentException: Illegal list shortcut value [date].
        at org.elasticsearch.painless.Def.lookupGetter(Def.java:454) ~[?:?]
        at org.elasticsearch.painless.DefBootstrap$PIC.lookup(DefBootstrap.java:149) ~[?:?]
        at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:203) ~[?:?]
        at org.elasticsearch.painless.PainlessScript$Script.execute(if (!doc['Time'].empty) {
return doc['Time'].date.hourOfDay + 7;
} return null;:45) ~[?:?]
        at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:133) ~[?:?]
        at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:51) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:164) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:422) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:393) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:390) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:644) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.2.jar:5.5.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_141]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_141]
4

Hi @warkolm
I think that error come from the logstash filter, because when i add if condition with tag, the file Time of filebeat become string type, not date type. How can i keep the data type when use if condition.
Here is my logstash filter

 if "node" in [tags] {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:Time} %{WORD:Method} %{NUMBER:Http_stt} %{NUMBER:ResTime}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    remove_field => "message"
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
5

It's hard to say without seeing the full error.

6

I said that because i check the field Time filtered type is String so the paintless scripts not work but when i use paintless script for @timestamp, it's working without any error

7

Oh, then yes that makes sense.

8

@warkolm so can you tell my why when i added the [tags] condition, all my field become string type? even the grokfilter is %{TIMESTAMP_ISO8601}
Thanks

9

Probably because you are doing a date filter on a different field, syslog_timestamp and not Time as per the grok.

10

Here is my grokfilter

 if "node" in [tags] {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:Time} %{WORD:Method} %{NUMBER:Http_stt} %{NUMBER:ResTime}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    remove_field => "message"
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }

but the type of Time field is string. It's because of the line ?

date {
          match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }

Thanks

11

Yeah, but syslog_timestamp doesn't exist in your grok.

12

Hi @warkolm
so instead of using

match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]

i should use

date {
match => [ "Time", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}

? or just remove the date filter
Thanks

13

Just update the date filter and it should be better.

1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.