ES 2 Courier Fetch: N of N shards failed

Elastic Stack Elasticsearch
1

Hi,

I get that message when i search in dashboard.
Mi config for threadpool is:

threadpool.search.queue_size: 11000

In kibana with google developer tools i get this response:

{
"responses": [
{
"_shards": {
"failed": 69,
"failures": [
{
"index": "logstash-2017.11.10",
"node": "X-BMD4EGTqi7KRKFaaCPGw",
"reason": {
"reason": null,
"type": "class_cast_exception"
},
"shard": 0
},
{
"index": "logstash-2017.11.11",
"node": "2jRReScrTq2i3gRcql28gg",
"reason": {
"reason": null,
"type": "class_cast_exception"
},
"shard": 0
}
],
"successful": 55,
"total": 124
},
"aggregations": {
"2": {
"buckets": []
}
},
"hits": {
"hits": [],
"max_score": 0.0,
"total": 0
},
"timed_out": false,
"took": 436
}
]
}

I don't actually know why this is happening. Also, in one of the views of the dashboard i get no data.

Please, could you help?

Thanks

2

Which version of Elasticsearch are you using? Also could you paste the request JSON that Kibana is sending too?

3

Version 2

IS this what you mean?

{"index":"logstash-*","search_type":"count","ignore_unavailable":true}
{"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":"host:mgaviewer AND (service:cassandra_read_latency OR service:cassandra_write_latency)"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"gte":1511524401065,"lte":1511528001065,"format":"epoch_millis"}}}],"must_not":[]}}}},"size":0,"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"1m","time_zone":"Europe/Berlin","min_doc_count":1,"extended_bounds":{"min":1511524401063,"max":1511528001063}},"aggs":{"3":{"terms":{"field":"service","size":10,"order":{"1":"desc"}},"aggs":{"1":{"max":{"field":"ms"}}}}}}}}

4

Yes that is what I mean thanks. Unfortunately the request doesn't show any hint of where the problem lies. Could you reproduce the error and then look in the Elasticsearch server log file to find the corresponding stack trace that will be in thee (it will contain ClassCastException and paste the whole stack trace (i.e. all the lines of the stack trace) here please? This should give us a better idea of where the problem is.

Also when you say you are running version 2 do you mean 2.0.0 or some other version within the 2.x.y series?

5

Hi colings86,

Yes i ment elasticsearch 2.0.0
This is the trace in elasticsearch log:

[2017-11-28 15:28:36,215][DEBUG][action.search.type ] [myservername] [logstash-2017.11.21][2], node[Jm8U1mBoTVGWPb8BI4XPNQ], [P], v[27], s[STARTED], a[id=TFDBmfAfTZig0-KG3TAB6g]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@513694ec] lastShard [true]
RemoteTransportException[[myservername][192.168.23.148:9300][indices:data/read/search[phase/query]]]; nested: ClassCastException;
Caused by: java.lang.ClassCastException

Hope that helps

6

@marcmga there should be more lines of the stack trace in the log file below the line you pasted which will indicate where in the code this exception was thrown. These line will start with at ... or caused by: .... Could you pasted all the lines below this log line up to the next line which starts [2017-11-28.....

7

Hi colings86,

no there's only this trace, repeated several times.
example:
[2017-11-30 12:28:57,507][DEBUG][action.search.type ] [srvjira] [logstash-2017.11.22][4], node[Jm8U1mBoTVGWPb8BI4XPNQ], [P], v[24], s[STARTED], a[id=Sl9a-3VETSeuuZPJel6jsw]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@1aafcd99] lastShard [true]
RemoteTransportException[[srvjira][192.168.23.148:9300][indices:data/read/search[phase/query]]]; nested: ClassCastException;
Caused by: java.lang.ClassCastException
[2017-11-30 12:28:57,500][DEBUG][action.search.type ] [srvjira] [logstash-2017.11.12][3], node[Jm8U1mBoTVGWPb8BI4XPNQ], [P], v[33], s[STARTED], a[id=ZHs42Q5AQqeFwtRfFqu6Rw]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@1aafcd99] lastShard [true]
RemoteTransportException[[srvjira][192.168.23.148:9300][indices:data/read/search[phase/query]]]; nested: ClassCastException;
Caused by: java.lang.ClassCastException
[2017-11-30 12:28:57,496][DEBUG][action.search.type ] [srvjira] [logstash-2017.11.26][2], node[Jm8U1mBoTVGWPb8BI4XPNQ], [P], v[2], s[STARTED], a[id=zcQE5sHdTw-_aDErwJG_eQ]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@1aafcd99] lastShard [true]
RemoteTransportException[[srvjira][192.168.23.148:9300][indices:data/read/search[phase/query]]]; nested: ClassCastException;
Caused by: java.lang.ClassCastException
[2017-11-30 12:28:57,496][DEBUG][action.search.type ] [srvjira] [logstash-2017.11.22][2], node[Jm8U1mBoTVGWPb8BI4XPNQ], [P], v[24], s[STARTED], a[id=1j2XmJzVQ0mftvWyFG3UYw]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@1aafcd99] lastShard [true]
RemoteTransportException[[srvjira][192.168.23.148:9300][indices:data/read/search[phase/query]]]; nested: ClassCastException;
Caused by: java.lang.ClassCastException
[2017-11-30 12:28:57,508][DEBUG][action.search.type ] [srvjira] [logstash-2017.11.13][3], node[Jm8U1mBoTVGWPb8BI4XPNQ], [P], v[33], s[STARTED], a[id=jkX2TStoSRy56-SQOQaKmA]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@1aafcd99]
RemoteTransportException[[srvjira][192.168.23.148:9300][indices:data/read/search[phase/query]]]; nested: ClassCastException;
Caused by: java.lang.ClassCastException
[2017-11-30 12:28:57,501][DEBUG][action.search.type ] [srvjira] [logstash-2017.11.21][3], node[Jm8U1mBoTVGWPb8BI4XPNQ], [P], v[24], s[STARTED], a[id=IUtBi6NgTCiQxUhvgrkiKA]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@1aafcd99] lastShard [true]
RemoteTransportException[[srvjira][192.168.23.148:9300][indices:data/read/search[phase/query]]]; nested: ClassCastException;
Caused by: java.lang.ClassCastException
[2017-11-30 12:31:26,920][DEBUG][action.search.type ] [srvjira] [logstash-2017.11.10][0], node[Jm8U1mBoTVGWPb8BI4XPNQ], [P], v[38], s[STARTED], a[id=RR4_wVoTR0qVw20P_jUCng]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@4cdb78de] lastShard [true]
RemoteTransportException[[srvjira][192.168.23.148:9300][indices:data/read/search[phase/query]]]; nested: ClassCastException;
Caused by: java.lang.ClassCastException
[2017-11-30 12:31:26,920][DEBUG][action.search.type ] [srvjira] [logstash-2017.11.13][0], node[Jm8U1mBoTVGWPb8BI4XPNQ], [P], v[33], s[STARTED], a[id=gNaQdh7BTxSJztLKcE7dAw]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@4cdb78de] lastShard [true]
RemoteTransportException[[srvjira][192.168.23.148:9300][indices:data/read/search[phase/query]]]; nested: ClassCastException;
Caused by: java.lang.ClassCastException

Where may i search for additional information?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.