Upgrade to Kibana and Elasticsearch 5

[vitgou]

Hi there,

I'm trying to upgrade to Kibana and Elasticsearch 5. I followed all the guides and when I check the health of the cluster I get:

{
"cluster_name" : "ELK",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 2996,
"active_shards" : 5992,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

But when I try to access kibana I get the following error:

[2017-01-05T19:44:47,555][WARN ][r.suppressed ] path: /logstash-/_field_stats, params: {allow_no_indices=false, index=logstash-, fields=*}
java.lang.ArrayIndexOutOfBoundsException: 5
at org.apache.lucene.util.UnicodeUtil.UTF8toUTF16(UnicodeUtil.java:602) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.apache.lucene.util.BytesRef.utf8ToString(BytesRef.java:152) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
at org.elasticsearch.action.fieldstats.FieldStats$Text.getMaxValueAsString(FieldStats.java:492) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.fieldstats.FieldStats$Text.toInnerXContent(FieldStats.java:498) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.fieldstats.FieldStats.toXContent(FieldStats.java:243) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.rest.action.RestFieldStatsAction$1.buildResponse(RestFieldStatsAction.java:87) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.rest.action.RestFieldStatsAction$1.buildResponse(RestFieldStatsAction.java:74) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:38) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:91) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:87) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.TransportAction$ResponseFilterChain.proceed(TransportAction.java:215) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.ingest.IngestActionFilter.apply(IngestActionFilter.java:89) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.TransportAction$ResponseFilterChain.proceed(TransportAction.java:213) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:240) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:226) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction.finishHim(TransportBroadcastAction.java:262) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction.onOperation(TransportBroadcastAction.java:215) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction$1.handleResponse(TransportBroadcastAction.java:195) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction$1.handleResponse(TransportBroadcastAction.java:182) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:978) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1055) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1039) [elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1029) [elasticsearch-5.1.1.jar:5.1.1]
at

I cannot understand if the issue is with Kibana + elasticsearch or If I messed up something during the upgrade.

Any idea?

Thanks
VG

[warkolm (Mark Walkom)]

Upgrade from what version?

[vitgou]

I upgraded elasticsearch from 2.4 to 5.1.

[Szymon_Szypulski]

I've had similar issue on one of my environments. It's due to output of /logstash-*/_field_stats query (as stated in logs). To confirm, run curl http://0.0.0.0:9200/logstash-*/_field_stats -d '{"fields":["*"],"allow_no_indices":false}'. It should crash with same error message, you can iterate over list of your indices (curl http://0.0.0.0:9200/_cat/indices?v) and run curl http://0.0.0.0:9200/logstash-<date>/_field_stats -d '{"fields":["*"],"allow_no_indices":false}' (replace <date>). Most of indices should return proper stats, if you find one which throws an error you can close the index curl http://0.0.0.0:9200/logstash-<data>/_close. Now you should be able to load kibana.

I don't know why this is happens and this is only solution I've found, It was possible to disable failing index because it was close to being deleted by our retention policy anyway.