Some indices broken after updating Kibana and Elasticsearch

[Dmytro_Orel]

Hi there,
I recently updated Kibana from 4.5 to 5.4.1 and Elasticsearch from 2.3 to 5.4.1. After this some indices start look broken:

1. in 'Discover' I get '"field" is a required parameter' message and it stays in 'Searching' state even if I use time range so small there are no entries in it;
2. existing visualizations and dashboards don't show any data and I get the same message as in previous item; when I try to add new visualization, it does not recognize any string/number/timestamp field, only geo_point;
3. if I open corresponding index pattern, no fields are searchable or aggregatable (except geo_point); refreshing field list does not help;
4. if I create new index through API, the result is the same as in item 3 (does not matter what fields it contains); I use nuget libraries Elasticsearch.Net and NEST, v5.4.0 both; and yes, I use Keyword/Text attributes.

There are the indices that work correct however. The only difference that I see between them is that those that stopped working contain geo_point field (but this field is the last thing working for them).

If it's the wrong category, please move it to the correct one.

Thanks in advance!

[Brandon_Kobel]

Hey @Dmytro_Orel have you tried refreshing your index patterns? To do so, if you click "Management" -> "Index Patterns"

and then select your index pattern to the left and click "Refresh fields list"

[Dmytro_Orel]

hi,
as I said in item 3, refresh fields didn't help.

[Brandon_Kobel]

@Dmytro_Orel would you mind describing the upgrade process that you followed?

[Dmytro_Orel]

sorry for the delay.
a standard procedure, described in docs: update debian repo definition, stop, apt-get install, start, first for Elasticsearch, than for Kibana. since we have only 1 VM, not a cluster, i thought it's not needed to disable shard allocation.
before updating Elasticsearch i installed migration helper plugin and checked that everything is green, except notification that from/to won't be available for geo_point any more (we don't use from/to anyway).

[Brandon_Kobel]

@Dmytro_Orel I'm sorry to tell you that this is being caused by https://github.com/elastic/elasticsearch/issues/24275 which is unresolved at this time. The only work-around that we have at this moment is to reindex your data, or wait for Kibana 5.5 to be released which won't rely on the _field_stats to determine index patterns.

[Dmytro_Orel]

just a basic reindex like first example in guide, or something more sophisticated?

and by the way, how will it fix issue with creating new index? just to remind:
4. if I create new index through API, the result is the same as in item 3 (does not matter what fields it contains); I use nuget libraries Elasticsearch.Net and NEST, v5.4.0 both; and yes, I use Keyword/Text attributes.

[Brandon_Kobel]

@Dmytro_Orel the basic reindex like the first example in the guide will work.

Regarding item 4, if you create a new index using the API and then created an index pattern in Kibana using the exact index name without wildcard names, what are you seeing?

[Dmytro_Orel]

here's the result:

UPD: fixed after i posted some dummy data there.
proceeding with reindex then.

[Brandon_Kobel]

@Dmytro_Orel Interesting... would you mind creating a basic index using CURL to determine if this is being caused by the client libraries or Elasticsearch itself? Does this happen when creating indices without the geo fields?

[Dmytro_Orel]

the screenshot is for index w/o geo fields (all of the fields are on screenshot actually).
same happens with CURL:

curl -XPUT 'localhost:9200/test2?pretty' -H 'Content-Type: application/json' -d'
{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"type1" : {
"properties" : {
"field1" : { "type" : "text" },
"field2" : { "type" : "date" }
}
}
}
}
'

[Dmytro_Orel]

reindex to new did the trick, thank you so much!