Kibana returns no data. Error: 70 of 2425 shards failed

[dorinand]

I encounter an error, when I trying to show messages from my index pattern. After short time, I am receiving error:

70 of 2425 shards failed

This message cant be enlarged, I can't click on it so basically, I do not know what is the core issue. I googled little bit and find out, it could be an issue with mapping or with shards. I check it, all my indices are in green state. All shards are in STARTED state.

I implement script, to query every index with:

curl -s 192.168.24.117:9200/$index/_search

I received no errors, an I also check the number of documents returned from /_cat/indices and $index/_search, they are matched.

I checked logs but when I perform request I can see no error in ES or Kibana.

I have approximately 550 indices, their mappings are created dynamically during index creation.

I create assumption that the problem could be with mappings (not sure). Can anybody helped me to find and fix damaged indices?

Kibana version: 6.7.1
Elastic version: 6.7.1

[dorinand]

After a day of investigation, I found a solution.

Because I was not able to find any error in Kibana or Eeasticsearch, I decide to use google chrome DevTools -> Network tool. In response data from Kibana, there were 14 indices with type and reason of error:

reason: "Trying to retrieve too many docvalue_fields. Must be less than or equal to: [100] but was [172]. This limit can be set by changing the [index.max_docvalue_fields_search] index level setting."
type: "illegal_argument_exception"

So I change settings for all indices from response and update max_docvalue_fields_search:

PUT /<index-name>/_settings
{
	"index.max_docvalue_fields_search" : "100000"
}

After this change, I am able to use Kibana again without any issue.

[wylie (Wylie Conlon)]

This is fixed in the future release of 7.11: https://github.com/elastic/kibana/issues/22897