My project is hitting a '5 of 30 shards failed' error when searching from Kibana. This is very similar to these postings:
Inspecting the browser requests, we do find:
"reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [ts] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
We have confirmed that we have multiple components of the system sending logs where sometimes the 'ts' field is a date and sometime it is text type. We would expect it to result in the
ts field not being searchable, that would be 'good', but because it's triggering this "shards failed" error the kibana queries just fail and return no logs. It seems like somehow ES is using different mappings on different shards exactly like was described in the above links.... but that should have been fixed in ES v2.x (the 2.x pull request was even linked in the second one) and we're on v5.5.1.
The problem is not consistent -- we have one system where we can see that the 'ts' value is not always the same type, the Kibana UI shows an "!" icon which expands to say:
Field Type Conflict: The type of this field changes across indices. It is unavailable for many analysis functions. The indices per type are as follows:
Field Type Index Names
This would be 'fine'/'good'.
But on another system this does not happen and instead the Kibana UI just says its a 'date' type (no "!" icon) but we get "shards failed" errors and cannot see the logs at all as a result ('bad').
How can we investigate the root cause further and find a solution?